[SERVER-17984] TTL Monitor doesn't modify query to match partial Index filter Created: 10/Apr/15  Updated: 19/Sep/15  Resolved: 08/May/15

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 3.1.1
Fix Version/s: 3.1.3

Type: Bug Priority: Major - P3
Reporter: Michael Grundy Assignee: J Rassi
Resolution: Done Votes: 0
Labels: 32qa, FT
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-19374 TTL Monitor does not yield while dele... Closed
is related to SERVER-18105 add ns to error log for bad ttl index... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

var t = db.index_filtered_ttl;
t.drop();
 
t.ensureIndex({x:1}, {expireAfterSeconds:2000, filter:{z:{$exists:1}}} )
 
var now = (new Date()).getTime();
var past = new Date(now - (3600 * 1000 * 2));
t.insert( { x : past, z:2} );
t.insert( { x : past} );
 
printjson(t.validate().keysPerIndex);
assert.eq(1,t.find({$query: {z:{$exists:1}}, $hint:{x:1}}).itcount(),
          "Wrong number of documents in partial index, before ttl monitor run");
assert.eq(2,t.find({}).itcount(),
          "Wrong number of documents in database, before ttl monitor run");
 
// wait for the ttl monitor to run
sleep(70 * 1000);
printjson(t.validate().keysPerIndex);
assert.eq(0,t.find({$query: {z:{$exists:1}}, $hint:{x:1}}).itcount(),
          "Wrong number of documents in partial index, after ttl monitor run");
assert.eq(1,t.find({}).itcount(),
          "Wrong number of documents in database, after ttl monitor run");

Sprint: Quint Iteration 3
Participants:

 Description   

When you create a partial TTL index the TTL Monitor uses the index key to perform the deletes. This results in documents that have the key value, but don't meet the filter criteria (therefore not indexed in the partial index), being deleted.



 Comments   
Comment by Githook User [ 08/May/15 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-17984 doTTLForIndex() delete with manual index scan
Branch: master
https://github.com/mongodb/mongo/commit/3a8bcdfb23ab85b57e5da308fb4d3c607ce77a49

Comment by Githook User [ 28/Apr/15 ]

Author:

{u'username': u'mgrundy', u'name': u'Mike Grundy', u'email': u'michael.grundy@10gen.com'}

Message: SERVER-17984: jstest for ttl partial index (currently disabled).
Branch: master
https://github.com/mongodb/mongo/commit/7c15941e120f0c72d66a2f13c21b6f59a64a492d

Comment by J Rassi [ 10/Apr/15 ]

Per discussion with Scott and Mathias:

  • TTL+partial will remain allowed (as this is a use case that we apparently want to support).
  • When creating the predicate for the delete request, the TTL monitor will extract the partial index document filter and AND it with the "expired" condition.
  • For performance reasons, we will consider hinting the TTL index when issuing the deletion query.
Comment by J Rassi [ 10/Apr/15 ]

Scott: in general, hinting an index shouldn't change the result set for a query. If it could, the query planner should be disallowing the hinted index for that query.

Comment by Scott Hernandez (Inactive) [ 10/Apr/15 ]

The problem here, IMHO, is really that our TTL deletes are just a query again any index right now, not the index the TTL index it is defined on.

Theoretically we could be seeing poor performance/behavior before due to this, where multiple indexes exist that meet the query produced during the TTL run.

Comment by J Rassi [ 10/Apr/15 ]

To me, this seems like a good argument for making TTL+partial illegal.

Comment by Scott Hernandez (Inactive) [ 10/Apr/15 ]

Another way to describe this, which is maybe more succinct or correct, is:
It is possible that the query planner may choose to delete things from the collection using an index other than the one which defined as the TTL index being used. We should hint/force the ttl index usage during the delete.

Generated at Thu Feb 08 03:46:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.