[DOCS-11661] Add a warning to discourage people from setting failIndexKeyTooLong Created: 27/Apr/18  Updated: 30/Oct/23  Resolved: 05/Oct/18

Status: Closed
Project: Documentation
Component/s: Atlas, Server
Affects Version/s: None
Fix Version/s: Server_Docs_20231030

Type: Improvement Priority: Major - P3
Reporter: Dmitry Ryabtsev Assignee: Jonathan DeStefano
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-35218 different result when query hint hint... Closed
Participants:
Days since reply: 5 years, 18 weeks, 5 days ago
Epic Link: DOCSP-2982
Story Points: 0.5

 Description   

We have witnessed a number of cases when live migrations into Atlas were failing because the users had documents that violate the index key size limit (1024 Bytes). The users have also confirmed that they run MongoDB with failIndexKeyTooLong (so they could insert such documents).

One part of the problem is that people are getting poor advice to set that parameter on sites like StackOverflow.

The other part of the problem is that the documentation section that talks about failIndexKeyTooLong isn't really discouraging people from setting it unless there is a good reason to do so. I think we could fix it by adding a stronger message there.



 Comments   
Comment by Githook User [ 05/Oct/18 ]

Author:

{'name': 'jonathan', 'email': 'jonathan.destefano@10gen.com', 'username': 'jdestefano-mongo'}

Message: DOCS-11661 - Kay's feedback.
Branch: master
https://github.com/mongodb/docs/commit/f3ff0727fb965bba6cb9ffb84e1e1752a53bc7f3

Comment by Githook User [ 05/Oct/18 ]

Author:

{'name': 'jonathan', 'email': 'jonathan.destefano@10gen.com', 'username': 'jdestefano-mongo'}

Message: DOCS-11661 - Tech review feedback.
Branch: master
https://github.com/mongodb/docs/commit/7127ae2ae084dc3669f6ea572ee030e4d44e6ca9

Comment by Dmitry Ryabtsev [ 05/Oct/18 ]

Hi jonathan.destefano,

The change looks good to me.

Thanks!

Comment by Jonathan DeStefano [ 04/Oct/18 ]

Hey dmitry.ryabtsev,

Made the requested edit with a few minor readability tweaks:
https://github.com/mongodb/docs/pull/3436

Please review at your convenience.

 

Thanks & Regards,

Jon

Comment by Dmitry Ryabtsev [ 04/Oct/18 ]

I was OOO so noticed you request for review when the ticket was already closed. In the warning you have the following wording:

Leaving indexes in place which contain oversized keys is not recommended, because they may return incomplete results.

That isn't technically accurate as with failIndexKeyTooLong there will not be an index entry created in case if the key size exceeds the hardcoded threshold of 1024 bytes. A better wording could be:

Leaving indexes in place where there are documents with indexed field values that are bigger than the indexKeyLimit threshold is not recommended, because queries using those indexes may return incomplete results

Comment by Githook User [ 03/Oct/18 ]

Author:

{'name': 'Steve Renaker', 'email': 'steve.renaker@10gen.com', 'username': 'steveren'}

Message: DOCS-11661: Add a warning to discourage people from setting failIndexKeyTooLong (#3423)
Branch: master
https://github.com/mongodb/docs/commit/a0befac0d425c8ea8a3d8534d93c2201cc8fc593

Comment by Steve Renaker (Inactive) [ 02/Oct/18 ]

dmitry.ryabtsev can you do a tech review of the PRs for this ticket when you have a moment? Thanks.

https://github.com/mongodb/docs/pull/3423

https://github.com/10gen/cloud-docs/pull/447

Comment by Steve Renaker (Inactive) [ 26/Sep/18 ]

The two PRs for this ticket are ready for tech review. william.byrne when you have a moment can you take a look?

https://github.com/mongodb/docs/pull/3423

https://github.com/10gen/cloud-docs/pull/447

 

Comment by Steve Renaker (Inactive) [ 26/Sep/18 ]

William, thanks for the detailed response. The current docs plan is:

  • update the server manual entry for the failIndexKeyTooLong parameter to emphasize that it is intended as a temporary workaround, not a permanent solution to the problem of noncompliant index keys
  • update the Atlas Live Import tutorials to mention that if noncompliant index keys exist and failIndexKeyTooLong is set to true, the migration will fail, and link to the server manual page which explains the index key limit and the problem of missing documents in indexes with noncompliant keys

These updates should be ready for review soon. Thanks again for the help.

Comment by William Byrne III [ 25/Sep/18 ]

> can anyone confirm that setting failIndexKeyTooLong to false will by itself cause Live Migrate to fail?

The opposite is more true. 

If a collection
 - has a document with a field (or set of fields) where the length of the values in those fields (plus metadata) adds up to more than the index key limit (i.e. 1024),

  • then (in version 2.4 and later) if you try and create an index over that field/set of fields, it will fail with and Index key too long error ...
  • ... unless you had failIndexKeyTooLong set to false, which ignores that error and leave that document out of the index.

Customers in their own hosted environments for various reasons have had failIndexKeyTooLong set to false. When they used LiveMigrate to transfer these databases to Atlas (where this parameter was not set) one or more index creations would fail due to the presence of documents with fields that exceeded the key limit.

Users used not be able to set this parameter in their Atlas environments, and had to open a ticket with support to have the Cloud team set it for them. However, recent changes in the Atlas cluster configuration options under "Additional Settings" now give them the option to set failIndexKeyTooLong and several other previously restricted parameters.

It is still the opinion of at least one MongoDB indexes expert (who sits in my chair) that setting this parameter and thus having incomplete indexes is a bad idea, as queries that use those indexes can return fewer documents than the same query using another index or a table scan. See SERVER-34891 for more on that.

The best solution is for customers who currently have this parameter set to false is to upgrade to a version after 4.1.3 where SERVER-36278 has removed the index key length limit altogether. Note that this would require them to:

  • wait for version 4.2 to be released
  • recreate all indexes with long keys to add in the missing documents

The short term workarounds to ensure LiveMigrate succeeds are to either

  • clean their data and/or change their index definitions to stay under the limit
  • set failIndexKeyTooLong set to false in their Atlas environments.

I hope dmitry.ryabtsev will comment too, but I believe that the hoped for documentation change was to encourage the data cleaning/index changes approach over setting failIndexKeyTooLong, because of the incomplete indexes => inconsistent results issue discussed in SERVER-34891.

Comment by Steve Renaker (Inactive) [ 25/Sep/18 ]

Hi folks – can anyone confirm that setting failIndexKeyTooLong to false will by itself cause Live Migrate to fail? Just checking that no other conditions are necessary.

Comment by Brian Moss [ 27/Apr/18 ]

In addition to updating the core docs, it might be a good idea to add a note to the Atlas Docs in the Live Migration Prerequisites sections for Replica Sets and Sharded Clusters.

Generated at Thu Feb 08 08:03:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.