[SERVER-26287] createIndexes should confirm that index options are valid in respect to the underlying collection Created: 23/Sep/16  Updated: 25/Apr/22  Resolved: 19/Nov/21

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: None
Fix Version/s: 5.2.0

Type: Bug Priority: Major - P3
Reporter: James Wahlin Assignee: Gabriel Marks
Resolution: Done Votes: 0
Labels: execution_intern, neweng
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-61545 Remove feature compatibility version ... Closed
Duplicate
is duplicated by SERVER-43820 createIndex does not validate index o... Closed
Related
related to SERVER-53739 Validate all createIndexes options Closed
related to SERVER-52539 Specify input/output to createIndexes... Closed
related to SERVER-53765 Validate index entries at catalog layer Backlog
is related to SERVER-769 Validate top-level & index spec field... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

> db.createCollection("test", {capped:true, size: 102400})
{ "ok" : 1 }
> db.test.ensureIndex({foo:1}, {expireAfterSeconds: 10})
{
        "createdCollectionAutomatically" : false,
        "numIndexesBefore" : 1,
        "numIndexesAfter" : 2,
        "ok" : 1
}
> db.test.insert({foo:new Date()})
WriteResult({ "nInserted" : 1 })

2016-01-19T18:17:11.210+1100 I STORAGE  [TTLMonitor] failing remove on a capped ns test.test
2016-01-19T18:17:11.210+1100 E INDEX    [TTLMonitor] Error processing ttl index: { v: 1, key: { foo: 1.0 }, name: "foo_1", ns: "test.test", expireAfterSeconds: 10.0 } -- 10089 cannot remove from a capped collection

Sprint: Execution Team 2021-11-01, Execution Team 2021-11-15, Execution Team 2021-11-29
Participants:
Case:

 Description   

As reported in SERVER-769, it is possible to create a TTL index on a capped collection. This results in failed TTL document deletion once expiration is hit.

We should audit which index options are invalid for specific collection options and add validation so that createIndexes will fail if specified.



 Comments   
Comment by Githook User [ 18/Nov/21 ]

Author:

{'name': 'Gabriel Marks', 'email': 'gabriel.marks@mongodb.com', 'username': 'marksg07'}

Message: SERVER-26287 Disallow creation of TTL indexes on capped collections
Branch: master
https://github.com/mongodb/mongo/commit/ba0fe9ac83cb253655a5ef79a396dcd432dcd69e

Comment by Eric Milkie [ 05/Nov/21 ]

Yes, or just try it with the option, catch the new error we're adding, and then retry without the option. Then you wouldn't have to poll every collection state.

Comment by Gregory Noma [ 01/Nov/21 ]

Yes that would be the ideal solution. One thing to potentially note about this is that it would require checking the state of the collection to see if it is capped, rather than just examining the index options themselves.

Comment by Eric Milkie [ 29/Oct/21 ]

If possible, the ideal outcome is for mongomirror to omit the expireAfterSeconds regardless of source and destination version, rather than failing. The failing option would require a user to then drop and rebuild that index without TTL (if they still needed the index) on their source system.

Comment by Tim Fogarty [ 29/Oct/21 ]

I added mirror/tools to downstream changes so we have a ticket open for this now.

Comment by Tim Fogarty [ 29/Oct/21 ]

Yes we only use listIndexes to replicate indexes between clusters. What you outlined sounds like a good plan to allow round-trips with same-version migrations. We do support mixed versions, so that will still be an issue in that case. I think the most similar change we've had in the past was dropping support for geoHaystack indexes in 5.0. For that we detect any geoHaystack indexes when we list indexes so mongomirror will fail before starting the migration. I think that's what we should do here too. If we're going to fail, we prefer to fail early.

Comment by Gregory Noma [ 27/Oct/21 ]

The only work left to do for this ticket should be to disallow creating TTL indexes on capped collections, which our documentation explicitly calls out as being disallowed. Note that while creating a TTL index on a capped collection currently succeeds, the index won't actually be TTL.

tim.fogarty does mongomirror (or any other tools) use only listIndexes to take indexes from one node and build them on another, or is there any other source of index information it uses? My idea is to exclude the expireAfterSeconds field from any index specs when running listIndexes on a capped collection, since those indexes aren't really TTL anyway. That way tools like mongomirror can still round-trip existing "TTL" indexes on capped collections without any changes, at least as long as both the source and destination nodes are on a version that includes this change. Any thoughts on this idea? Would the mixed-version scenario still be an issue? (I'm not sure exactly what we support for mongomirror.) I guess the more general question is how we've dealt with changes like this in the past.

Comment by Eric Milkie [ 21/Oct/21 ]

Is this fix going to allow tools like mongomirror the ability to still build indexes that fail this check, or is there downstream work for all the tools to avoid bad index specs?

Generated at Thu Feb 08 04:11:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.