[SERVER-27243] Upgrade to 3.4 'create collection' fails in Backup Created: 30/Nov/16  Updated: 05/Apr/17  Resolved: 13/Dec/16

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 3.4.0
Fix Version/s: 3.4.2, 3.5.1

Type: Bug Priority: Major - P3
Reporter: Isabel Peters Assignee: Charlie Swanson
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File dataset.tgz     Text File logOutputVerbose.log    
Issue Links:
Backports
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.4
Sprint: Query 2016-12-12, Query 2017-01-23
Participants:

 Description   

After upgrading to 3.4. backup for a deployment stopped working. Investigation into the mongod logs shows following error while trying to apply oplogs.

m.xgen.svc.brs.util.GenericMongoManager$MongoManagerStartException: Upgrade command failed. Cmd: `[numactl, --interleave=all, /opt/mongodb/mongodb-linux-x86_64-rhel62-3.4.0/bin/mongod, --bind_ip=0.0.0.0, --noauth, --slowms=1000, --setParameter, diagnosticDataCollectionEnabled=false, --dbpath=/data/backups/daemon/510357167fe227e9f1832923/rs0/head/, --setParameter, ttlMonitorEnabled=false, --setParameter, failIndexKeyTooLong=false, --port=27503, --upgrade, --storageEngine=mmapv1, --nojournal, --setParameter, internalValidateFeaturesAsMaster=false]` Code: `134`
 
StdOut:
2016-11-29T03:22:17.130+0000 I CONTROL  [initandlisten] MongoDB starting : pid=22877 port=27503 dbpath=/data/backups/daemon/510357167fe227e9f1832923/rs0/head/ 64-bit host=brs27.nj1.10gen.cc
2016-11-29T03:22:17.131+0000 I CONTROL  [initandlisten] db version v3.4.0
2016-11-29T03:22:17.131+0000 I CONTROL  [initandlisten] git version: f4240c60f005be757399042dc12f6addbc3170c1
2016-11-29T03:22:17.131+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-11-29T03:22:17.131+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2016-11-29T03:22:17.131+0000 I CONTROL  [initandlisten] modules: none
2016-11-29T03:22:17.131+0000 I CONTROL  [initandlisten] build environment:
2016-11-29T03:22:17.131+0000 I CONTROL  [initandlisten]     distmod: rhel62
2016-11-29T03:22:17.131+0000 I CONTROL  [initandlisten]     distarch: x86_64
2016-11-29T03:22:17.131+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2016-11-29T03:22:17.131+0000 I CONTROL  [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 27503 }, operationProfiling: { slowOpThresholdMs: 1000 }, security: { authorization: "disabled" }, setParameter: { diagnosticDataCollectionEnabled: "false", failIndexKeyTooLong: "false", internalValidateFeaturesAsMaster: "false", ttlMonitorEnabled: "false" }, storage: { dbPath: "/data/backups/daemon/510357167fe227e9f1832923/rs0/head/", engine: "mmapv1", journal: { enabled: false } }, upgrade: true }
2016-11-29T03:22:17.295+0000 I -        [initandlisten] Fatal assertion 18523 InvalidOptions: The field 'create' is not a valid collection option. Options: { capped: true, create: "oauthnonces", flags: 1, size: 1048576 } at src/mongo/db/storage/mmap_v1/mmap_v1_database_catalog_entry.cpp 876
2016-11-29T03:22:17.296+0000 I -        [initandlisten]
 
***aborting after fassert() failure

The version for the first backup snapshot is 3.0.7 (2015). The deployment dates back to 2013 (when collection oauthnonces was first created).

Find the `oauthnonces` dataset attached.

CC: steve.briskin, john.morales



 Comments   
Comment by Githook User [ 22/Dec/16 ]

Author:

{u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'}

Message: SERVER-27243 Ignore invalid collection options if 'create' present

It is possible for older versions of MongoDB to save invalid collection
options which should be ignored. Older versions also would include a
'create' field in the collection options when a collection was created,
and possibly re-order the fields of the options document during a
subsequent 'collMod'. If the 'create' option is present at all, we
should take that to mean that these options were created on an older
version of MongoDB, and ignore unknown options. Otherwise, we should
reject unknown options.

(cherry picked from commit 68908f8e03ee1a4c0400dae26cc20b0c304b913b)
Branch: v3.4
https://github.com/mongodb/mongo/commit/c3bb0245b19541cab21a7fb712e34229b4cf9694

Comment by Githook User [ 13/Dec/16 ]

Author:

{u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'}

Message: SERVER-27243 Ignore invalid collection options if 'create' present

It is possible for older versions of MongoDB to save invalid collection
options which should be ignored. Older versions also would include a
'create' field in the collection options when a collection was created,
and possibly re-order the fields of the options document during a
subsequent 'collMod'. If the 'create' option is present at all, we
should take that to mean that these options were created on an older
version of MongoDB, and ignore unknown options. Otherwise, we should
reject unknown options.

(cherry picked from commit 6fba074768fad5f47611de38682257257005a1a6)
Branch: master
https://github.com/mongodb/mongo/commit/68908f8e03ee1a4c0400dae26cc20b0c304b913b

Comment by Charlie Swanson [ 06/Dec/16 ]

steve.briskin as discussed in person, our best guess is that the order of the collection options are getting jumbled somewhere between reading them from the oplog and being stored on the backup daemon's mongod. I have done some testing and also read the code for a while. I don't see any place where the order could get swapped server side. Would it be possible for you to do some testing and log what the command looks like at various points in the process (e.g. when interpreted in go driver, when sent to the backup agent, when sent as applyOps command from the backup agent)?

If you can't do this, is there somewhere you could point me to to set up the backup daemon myself and also point me where to add the logging (if you happen to know)?

On a separate note, I have also tested (at least against 2.4, I can't remember if I've tested on newer versions) what the server does if it receives an applyOps with the op being in the wrong order (e.g. {capped: true, create: "oauthnonces"} instead of {create: "oauthnonces", capped: true}). It looks like the server does not respond with an error, but neither does it actually create a collection with those invalid options. So I still have no clear picture of how this happened...

Comment by Asya Kamsky [ 05/Dec/16 ]

Python won't maintain order entered of a dictionary (so when order is important we recommend using SON class). It also used to allow passing in options dictionary (deprecated in pymongo 2.2) to create_collection command and in 2.4 server it would have just stored all the options passed in.

Comment by Tess Avitabile (Inactive) [ 01/Dec/16 ]

I wonder whether the driver is alphabetizing the options.

Comment by James Wahlin [ 01/Dec/16 ]

The 3.4.0 code base rejects unknown options. There is an exception for the "create" field which was generated in versions 2.4 and earlier but we expect it to be the first field in the options metadata (see this comment). It appears to be in the second position here:

Options: { capped: true, create: "oauthnonces", flags: 1, size: 1048576 }

Generated at Thu Feb 08 04:14:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.