[SERVER-46640] The TTLMonitor can remove the cached index information from the TTLCollectionCache for a newly created index before the implicitly created collection is registered in the CollectionCatalog Created: 05/Mar/20  Updated: 29/Oct/23  Resolved: 13/Mar/20

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 4.4.0-rc0, 4.5.1
Fix Version/s: 4.4.0-rc0, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Gregory Wlodarek Assignee: Gregory Wlodarek
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Steps To Reproduce:

Remove these lines in ttl.cpp: https://github.com/mongodb/mongo/blob/3e30e2f0705cbb167d28cfe6f5344c354f2e9442/src/mongo/db/ttl.cpp#L94-L97

Run `resmoke --suites=no_passthrough jstests/noPassthrough/ttl_partial_index.js`

It should fail on the first execution, if not, just run it again with --opt=on.

Sprint: Execution Team 2020-03-23
Participants:
Linked BF Score: 13

 Description   

The TTLMonitor uses the CollectionCatalog to look-up the collection namespace given a UUID here (prior to obtaining any collection locks). If the collection with the given UUID does not exist in the CollectionCatalog, the cached information gets removed from the TTLCollectionCache for that index. This will prevent the TTLMonitor from ever running on that index, and as a consequence, the TTLMonitor won't remove any expired documents in the collection for the given index unless it is recreated, which would cause it to be registered in the TTLCollectionCache again.

 

Based on these logs, when creating a new index that requires the collection to be created implicitly, the index will be registered in the TTLCollectionCache before the collection is registered with the CollectionCatalog. If the TTLMonitor were to run between these two events, it would think that the collection for that index was dropped, and so it will deregister that index from the TTLCollectionCache.

 

 

{"t":{"$date":"2020-03-05T09:08:09.268-0500"},"s":"I", "c":"STORAGE", "id":20320,"ctx":"conn1","msg":"createCollection: {nss} with {generatedUUID_generated_provided} UUID: {optionsWithUUID_uuid_get} and options: {options}","attr":{"nss":"test.ttl_partial_index","generatedUUID_generated_provided":"generated","optionsWithUUID_uuid_get":{"uuid":{"$uuid":"4a58143c-819c-43bd-890e-368d0389dc4d"}},"options":{}}}
{"t":{"$date":"2020-03-05T09:08:09.268-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}
...
{"t":{"$date":"2020-03-05T09:08:09.298-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}
{"t":{"$date":"2020-03-05T09:08:09.319-0500"},"s":"I", "c":"INDEX",   "id":20345,"ctx":"conn1","msg":"index build: done building index {indexName} on ns {nss}","attr":{"indexName":"_id_","nss":"test.ttl_partial_index"}}
{"t":{"$date":"2020-03-05T09:08:09.322-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}
...
{"t":{"$date":"2020-03-05T09:08:09.322-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}
{"t":{"$date":"2020-03-05T09:08:09.322-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"conn1","msg":"{}","attr":{"message":"+++: Registering in TTLCollectionCache { v: 2, key: { x: 1.0 }, name: \"x_1\", expireAfterSeconds: 0.0, partialFilterExpression: { z: { $exists: true } } }"}}
{"t":{"$date":"2020-03-05T09:08:09.322-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}
{"t":{"$date":"2020-03-05T09:08:09.322-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 1"}}
{"t":{"$date":"2020-03-05T09:08:09.322-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: UUID 4a58143c-819c-43bd-890e-368d0389dc4d indexName x_1"}}
{"t":{"$date":"2020-03-05T09:08:09.322-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: deregistering collection from ttlCollectionCache"}}
{"t":{"$date":"2020-03-05T09:08:09.322-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}
...
{"t":{"$date":"2020-03-05T09:08:09.352-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}
{"t":{"$date":"2020-03-05T09:08:09.352-0500"},"s":"I", "c":"INDEX",   "id":20345,"ctx":"conn1","msg":"index build: done building index {indexName} on ns {nss}","attr":{"indexName":"x_1","nss":"test.ttl_partial_index"}}
{"t":{"$date":"2020-03-05T09:08:09.352-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}
...
{"t":{"$date":"2020-03-05T09:08:09.352-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}
{"t":{"$date":"2020-03-05T09:08:09.352-0500"},"s":"I", "c":"STORAGE", "id":0,"ctx":"conn1","msg":"{}","attr":{"message":"+++: registering collection 4a58143c-819c-43bd-890e-368d0389dc4d test.ttl_partial_index"}}
{"t":{"$date":"2020-03-05T09:08:09.352-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}
...
{"t":{"$date":"2020-03-05T09:08:09.352-0500"},"s":"I", "c":"INDEX",   "id":0,"ctx":"TTLMonitor","msg":"{}","attr":{"message":"+++: length of ttlInfos: 0"}}



 Comments   
Comment by Githook User [ 13/Mar/20 ]

Author:

{'name': 'Gregory Wlodarek', 'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com'}

Message: SERVER-46640 The TTLMonitor should not remove the cached index information from the TTLCollectionCache when a collection is not yet visible

(cherry picked from commit df0272716ae3a6767e33adf98392dddea494e587)
Branch: v4.4
https://github.com/mongodb/mongo/commit/8f6fa99d2c9eb037db6bb9522cf0fb4ca1d938e0

Comment by Githook User [ 13/Mar/20 ]

Author:

{'name': 'Gregory Wlodarek', 'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com'}

Message: SERVER-46640 Register index information in TTLCollectionCache upon a successful index build

(cherry picked from commit 7ce09a820efeceb4f0a3a70140748aaf88309452)
Branch: v4.4
https://github.com/mongodb/mongo/commit/184d3641106093b10656490aaae107c426cef85f

Comment by Githook User [ 13/Mar/20 ]

Author:

{'name': 'Gregory Wlodarek', 'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com'}

Message: SERVER-46640 The TTLMonitor should not remove the cached index information from the TTLCollectionCache when a collection is not yet visible
Branch: master
https://github.com/mongodb/mongo/commit/df0272716ae3a6767e33adf98392dddea494e587

Comment by Githook User [ 13/Mar/20 ]

Author:

{'name': 'Gregory Wlodarek', 'username': 'GWlodarek', 'email': 'gregory.wlodarek@mongodb.com'}

Message: SERVER-46640 Register index information in TTLCollectionCache upon a successful index build
Branch: master
https://github.com/mongodb/mongo/commit/7ce09a820efeceb4f0a3a70140748aaf88309452

Generated at Thu Feb 08 05:12:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.