[SERVER-16450] Warn/error if invalid indexes present Created: 08/Dec/14  Updated: 07/Jun/17  Resolved: 07/Jun/17

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 2.8.0-rc2
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Kevin Pulo Assignee: Kevin Pulo
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-16628 Fix upgrade checker for 3.0 Closed
related to SERVER-17658 Upgrade/downgrade support for partial... Closed
Backwards Compatibility: Minor Change
Participants:

 Description   

In a replica set, initial syncs will fail if a collection contains an invalid index. This means that if such indexes are present from older versions (2.4 or earlier), they are "time bombs" that will prevent an initial sync at some later time. It would be better if these invalid indexes could be detected at startup, and a startup warning recorded (or some other appropriate behaviour).



 Comments   
Comment by Kevin Pulo [ 07/Jun/17 ]

As of 2.6, bad indexes such as {$a:1} or {"b..c":1} can no longer be created. This means that these indexes can only be created with 2.4 or earlier, which is now well EOL.

As of 3.2, if such bad indexes are present in any collection at startup, the server will fassert (28782) and refuse to start up (SERVER-17658).

If any such indexes are present and preventing upgrading to 3.2, the workaround is to start up on 3.0 again, drop the offending index(es), and then upgrade to 3.2.

Based on all of this, the need for the warning requested on this ticket has Gone Away.

Comment by Kevin Pulo [ 09/Dec/14 ]

I don't see any warnings, and checking the key spec should be sufficient. My exact steps are:

$ mlaunch --single --port 16450 --binarypath /sw/mongodb/2.4.12/bin
$ mongo --port 16450
> db.foo.insert({})
> db.foo.ensureIndex({$a:1})
> db.foo.ensureIndex({"b..c":1})
> db.foo.find()
> db.foo.getIndexes()
> exit
$ mlaunch restart --binarypath /sw/mongodb/2.6.5/bin
$ mongo ---port 16450
> db.version()
> show startupWarnings
> db.foo.find()
> db.foo.getIndexes()
> exit
$ mlaunch restart --binarypath /sw/mongodb/2.8.0-rc2/bin
$ mongo ---port 16450
> db.version()
> show startupWarnings
> db.foo.find()
> db.foo.getIndexes()
> exit
$ grep -i warn data/mongod.log
$ grep -i error data/mongod.log
$ grep -i invalid data/mongod.log
$ grep -i index data/mongod.log

The "show startupWarnings" and greps give no results, except for the last one which shows the index creations (under 2.4), but nothing after that.

I want to be able to run this sequence of commands, and under 2.6+ get some indication that my indexes are not kosher (well, no longer kosher).

For the initial sync issue, is that not already adequately covered by SERVER-16449? I understand what you're saying about not causing replication failures in the absence of user changes, but it seems to me that the change to disallow "$" and ".." indexes is a backward breaking change where we can't help but violate this (on standalones as well as in replsets/sharded).

Comment by Scott Hernandez (Inactive) [ 09/Dec/14 ]

We do validate all indexes at start-up and warn/error if invalid indexes are found, in 2.4, 2.6 and master. It might be that we need to validate more than just the key spec at start-up now, to match what ensure/createIndexes does. If that is what you want then let's create a new issue for that, with an example of what is not working, and close this one.

I think the initial sync one might be something else to file, which I would summarize as, "don't fail initial sync due to (index) validation rules", or something like that. I have a rule which I'd like to see all parts of the server follow, if possible: "data in the system should not fail to replicate nor shard if the user doesn't change it" – more or less it means that stored data should not cause problems (if possible) until the user, not system, tries to modify it.

Comment by Kevin Pulo [ 09/Dec/14 ]

Yes, all I have been asking is for master (and maybe 2.6) to check at startup (or db first use) if there are any invalid indexes present, and if so, log a startup warning.

The indexes should not be there in 2.6+, so 2.6+ servers should be vocal if any are found. I am not asking for them to be allowed in any way. I'm asking for them to be noticed earlier, so that an inadvertent user can be notified and deal with them. If you think there is something else which more appropriate for 2.6+ to do when it proactively encounters such indexes, then that would be fine too.

I have no problem with these indexes existing in older versions, since they don't barf on them. I have no problem with the idea that they shouldn't be in 2.6+. But it is trivial to have a 2.6+ instance which contains these indexes (just create them in an earlier version, and then upgrade without running upgradeCheck).

Comment by Scott Hernandez (Inactive) [ 09/Dec/14 ]

I think your idea isn't going to work very well from the server POV since you want it to warn you when running the old version, where the index is just fine...and where the new server version will have issues. If the old version knew that those indexes were going to a problem in the future, which we know now, and have "fixed" in newer server versions, then our past selves should have coded that up, now that we know what is acceptable.... basically, you want a back-port, to all existing/old server binaries, which is a problem given the existence of version which allow those types of indexes.

The reason we created the upgradeChecker in the shell, in a new version of the shell you could run against old server versions, before upgrading mongo server, was because we couldn't alter the existing server code which was already deployed.

Can you think of some way for your request to actually be implemented? Maybe you want the new server version to log a warning rather than not starting? In the face of replication and initial sync, maybe you want it to loosen the restrictions as well. If that is your idea, we can record it but I'm not sure how much traction it will get given in essence it comes down to "don't ensure valid options/indexes" in the new version, which is something we want and was changed on purpose to help everyone now and in the future.

Comment by Kevin Pulo [ 09/Dec/14 ]

You're right that this doesn't directly have anything to do with replication, which is just one of the more visible symptoms. There may be other latent problems that invalid indexes cause, I don't know. This is more about the fact that because the upgradeChecker is only advisory, it is possible to end up with invalid indexes. Since they are not valid indexes, they shouldn't be there, and we should let the user know that. Aside from the upgradeChecker (which is only opt-in), there's no visibility into the fact that the instance contains indexes which might cause problems later on down the track.

I'm just imagining something along the lines of checkForIdIndexes in SERVER-8238, i.e. just iterate through all indexes in all collections, and log a startup warning for any invalid ones that are found.

Comment by Scott Hernandez (Inactive) [ 08/Dec/14 ]

Can you provide a use-case and indicate which node should warn, and when? Does this really have anything to do with replication or can this be described just in terms of a stand-alone instance?

Generated at Thu Feb 08 03:41:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.