[SERVER-12983] Restore from 2.4 DB into 2.6 DB fails due to index? Created: 01/Mar/14  Updated: 20/Jan/16  Resolved: 02/Mar/14

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: 2.6.0-rc0
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Michael Kennedy Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File C__WINDOWS_system32_cmd.exe (53).png    
Issue Links:
Duplicate
duplicates SERVER-12982 Could not restore backup data due to ... Closed
Operating System: ALL
Steps To Reproduce:

Use this command and see screenshot.

mongorestore --drop --db DB ".\DB"
echo done

Participants:

 Description   

It appears that if a 2.4 DB which was backed up using MongoDump is then restored via MongoRestore into 2.6 and that dump contains an index which is too large to index, then the restore stops and leaves the DB incompleted.

Notice how the last line from MongoRestore is

Error creating index LearningLine.Conversation: 17282 err: "Btree::insert: key too large to index, failing LearningLine.Conversation.$find_messages_by_activity 1426 { : { Attachments: [], LastAttachmentId: 0, C..."

This is NOT the final collection to restore.



 Comments   
Comment by Daniel Doubrovkine [ 20/Jan/16 ]

Yup, opened https://jira.mongodb.org/browse/SERVER-22251, the 2.6 version of the shell at least seems to be doing something.

Comment by Daniel Pasette (Inactive) [ 20/Jan/16 ]

Pretty sure you've found a bug in the upgradeChecker in 3.2.1.

Can you use the 2.6 mongo shell and try again. Also, upgradeCheckAllDBs() is just a convenience method to do the iteration for you.

Comment by Daniel Doubrovkine [ 20/Jan/16 ]

Running 3.2.1 shell against a 2.4 MongoDB.

$ mongo ...
MongoDB shell version: 3.2.1
connecting to: ...
> db.upgradeCheck()
database 'gravity-staging' for 2.6 upgrade compatibility
...
Everything in 'gravity-staging' is ready for the upgrade!

^^^ lies

I cannot run upgradeCheckAllDBs() because this is a Compose deployment, I don't have admin access. I assume that it just iterates over and does upgradeCheck?

Comment by Daniel Pasette (Inactive) [ 20/Jan/16 ]

What exactly are you running for upgradeCheck? What version of the shell and mongodb are you using?
Are you following this documentation: https://docs.mongodb.org/manual/reference/method/db.upgradeCheckAllDBs?

Comment by Daniel Doubrovkine [ 20/Jan/16 ]

Finding these has been a painful trial/error process, mongodump, mongorestore, restart once the next one fails.

db.upgradeCheck() returned nothing

The error from the index creation from mongoimport just leaves you guessing, hunting for the document ID that causes the problem. It would be nice it it at least returned the ID of the document - the one below for example doesn't get found with a regex because it's not in the index (partial index)?

Failed: gravity-staging.events: error creating indexes for gravity-staging.events: createIndex error: Btree::insert: key too large to index, failing gravity-staging.events.$name_1 6590 { : "Activities New York / Description Soonja Han's Paris Studio gets reconstructed within the de Buck gallery in Chelsea, NY: An interrogation of the ar..." }

Comment by Daniel Pasette (Inactive) [ 20/Jan/16 ]

Hi DB,
Sorry you hit this, but as you can see this repairs a major correctness issue. Starting with 2.6, if the index key is too long, the document is rejected. The change is described in this ticket: SERVER-5290. If the run parameter failIndexKeyTooLong is set to false, the document will be inserted, but the index entry will not be created.

This is documented here:
https://docs.mongodb.org/manual/reference/parameters/#param.failIndexKeyTooLong

Comment by Daniel Doubrovkine [ 20/Jan/16 ]

FWIW I spent the majority of the afternoon working around this error, collection after collection. We query data by the problematic indexe(s) so this restriction and the inability to ignore and continue came as a surprise on a mongorestore. Being able to run the server in an old(er) mode is fine, but we don't run our own, I would have liked to get (or be able to get) a warning instead of a hard error here.

Related, I couldn't find in the documentation anything about the semantics of inserting a record when the index key produced would be too long - does the record get inserted or not?

Comment by Michael Kennedy [ 03/Mar/14 ]

Hi Dan,

Yes, that is what my understanding was. I filed the bug because I think this is wrong.

It's fine for runtime. But I want a flag or something for *mongorestore* which says something like 'continue on index error' or 'drop indexes that are incompatible with 2.6' or something. I do not want to run the server is a downgraded mode. I just want to import the data, acknowledging the problems and changes.

Comment by Daniel Pasette (Inactive) [ 03/Mar/14 ]

With the new restrictions on index key length, you have a couple options to restore databases with such documents.
1. restore with a 2.4 server, then restart with 2.6 binary.
2. run with server with the failIndexKeyTooLong set to false to override the new behavior.

In either case, it is highly recommended that the documents are cleaned up to avoid condition if possible as results from indexes will not be usable for docs with this keys that are too long.

Comment by Michael Kennedy [ 02/Mar/14 ]

Thanks for looking at this Dan. I'm not sure I would classify it as duplicate.

I think there is a difference between attempting a DB restore and attempting to create an improper index and runtime or doing an insert which is invalid. I don't want to run the server in a degraded mode, I just want to restore data and acknowledge any index errors (but continue)

Comment by Daniel Pasette (Inactive) [ 02/Mar/14 ]

duplicate of SERVER-12982.

Generated at Thu Feb 08 03:30:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.