[DRIVERS-774] Deprecate capped collections and use improved TTL as replacement Created: 21/Nov/19  Updated: 06/Jun/23  Resolved: 06/Jun/23

Status: Closed
Project: Drivers
Component/s: Performance
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Backlog - Core Eng Program Management Team Assignee: Unassigned
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends

 Description   
Downstream Change Summary

null

Description of Linked Ticket

Epic Summary

Problems

  • Capped collections do not replicate properly. They do not replicate record deletes, which can result in data corruption between replica set nodes (documents present on some nodes and not others). Due to the design of mmapV1 capped collections, it is not possible to replicate capped collection deletes for mmapV1.
  • Capped collection performance is not good, despite one of the purported use cases being "to get good performance from an append-only workload".

Goal

  • Deprecate capped collections and replace them with something that doesn't corrupt replicated data

Nongoal

  • The special Oplog capped collection does not suffer from the data corruption issue, and already has custom code to make it unlike all other capped collections. It does not need to be converted to a TTL collection.

Possible Implementation Details

  • Allow TTL index by specifying number of documents or data size as expiration criteria
  • Create logOp for TTL expiration specifying a field and expiration point combo
  • Allow TTL index on _id
  • Make TTL collections with TTL on _id use clustered index
  • Efficiently truncate TTL collections with TTL on _id

Generated at Thu Feb 08 08:22:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.