[SERVER-19334] TTL index deletions cannot always keep up with insertions with WiredTiger Created: 08/Jul/15 Updated: 06/Dec/22 Resolved: 02/Apr/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance, Storage, WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Eitan Klein | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Incomplete | Votes: | 1 |
| Labels: | 32qa | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Assigned Teams: |
Storage Execution
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Sprint: | Integration F (02/01/16), Integration 10 (02/22/16) | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Configuration: db version v3.1.6-pre- Environment
Workload:
Issues: MongoDb reported that document has been deleted w/ TTL working well on the same workload , same build w/ MMAP storage engine Additional informationPlease see |
| Comments |
| Comment by Githook User [ 16/Oct/15 ] | |||||||||||||
|
Author: {u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}Message: | |||||||||||||
| Comment by Daniel Pasette (Inactive) [ 03/Aug/15 ] | |||||||||||||
|
re-opening this issue to pursue approach of batching deletes | |||||||||||||
| Comment by Geert Bosch [ 13/Jul/15 ] | |||||||||||||
|
The issue is that restoreState resets the timer, and we call this function in the delete loop of doTTLForIndex:
As a result the TTL monitor never yields. | |||||||||||||
| Comment by Daniel Pasette (Inactive) [ 13/Jul/15 ] | |||||||||||||
|
Broke out the separate issue found while investigating this one here: | |||||||||||||
| Comment by Daniel Pasette (Inactive) [ 10/Jul/15 ] | |||||||||||||
|
This did expose a possible issue with the TTL monitor though. When the TTL monitor is running, I found that listCollections would block. Easy to repro:
| |||||||||||||
| Comment by Daniel Pasette (Inactive) [ 10/Jul/15 ] | |||||||||||||
|
Depending exactly on the parameters used in the test, it is very easy to outstrip the ability of mongod to delete as much as what is being inserted. The TTL monitor runs 1/minute and is single threaded, deleting each document 1 at a time. This can be exacerbated with a small cache, which will force the deleted documents to be pulled off disk. With MMAP, I believe the insertion rate is naturally throttled by the deletions because of the collection level lock. This explains why the data would take a long time to be deleted. The disk space then would only be reclaimed gradually by new data, but without an explicit compact, I don't think it would be returned to the system, and thus would stay at the high water mark it reached during the insertion phase. I don't think there is a ton we can do right now to make this much better, other than possibly parallelizing the deletes in some manner. | |||||||||||||
| Comment by Sam Kleinman (Inactive) [ 08/Jul/15 ] | |||||||||||||
|
The TTL process only deletes documents every 60 seconds, and the configurability of the interval that the TTL deletion operation is not documented (noted as "for testing only".) https://github.com/mongodb/mongo/blob/master/src/mongo/db/ttl.cpp#L78 The TTL documentation states that expired documents may continue to exist in the database for some time after they've reached the timeout. Thus this is expected behavior. Cheers, | |||||||||||||
| Comment by Eitan Klein [ 08/Jul/15 ] | |||||||||||||
|
Easy repro script db.foo.createIndex( {date: 1}, {expireAfterSeconds: 20}) ) } |