[SERVER-33494] WT SizeStorer never deletes old entries Created: 26/Feb/18  Updated: 20/Nov/23  Resolved: 13/Nov/23

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 7.3.0-rc0, 7.2.0-rc2, 7.0.5, 6.0.13, 5.0.24

Type: Bug Priority: Major - P3
Reporter: Daniel Gottlieb (Inactive) Assignee: Gregory Noma
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-82902 Dump size storer table in JSON format... Blocked
is depended on by SERVER-82903 Filter size storer entires using WT t... Blocked
Problem/Incident
Related
is related to SERVER-83120 Mechanism for one-time cleanup of orp... Backlog
is related to SERVER-83336 Temporarily disable wt_size_storer_cl... Closed
Assigned Teams:
Storage Execution
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.2, v7.0, v6.0, v5.0
Sprint: Execution Team 2023-11-13, Execution Team 2023-11-27
Participants:
Case:
Linked BF Score: 152

 Description   

The WTSizeStorer is used to keep fast counts for each collection in the system. Creating a collection adds an `ident` -> `count` mapping, but those key/value pairs are never deleted.

RTT makes this problem worse. When rolling forward from the stable point -> common point re-creates a collection, it will get a new `ident`, thus leaking the previous ident in the table.



 Comments   
Comment by Githook User [ 16/Nov/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-33494 Remove size storer entry upon ident drop

(cherry picked from commit b4e5daaeb51920b0074661c87dc3b3765431e04e)
Branch: v5.0
https://github.com/mongodb/mongo/commit/529707336a6f57dfced28223dccc14eb5ac38d6d

Comment by Githook User [ 16/Nov/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-33494 Remove size storer entry upon ident drop

(cherry picked from commit b4e5daaeb51920b0074661c87dc3b3765431e04e)
Branch: v7.2
https://github.com/mongodb/mongo/commit/7c9b028298a072b3cca5306929a4f82cf2768d77

Comment by Githook User [ 16/Nov/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-33494 Remove size storer entry upon ident drop

(cherry picked from commit b4e5daaeb51920b0074661c87dc3b3765431e04e)
Branch: v6.0
https://github.com/mongodb/mongo/commit/ee5f8351c194c60f58e85d1f37bec107fde26301

Comment by Githook User [ 15/Nov/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-33494 Remove size storer entry upon ident drop

(cherry picked from commit b4e5daaeb51920b0074661c87dc3b3765431e04e)
Branch: v7.0
https://github.com/mongodb/mongo/commit/d8f2a0cbc53ad9d689ef8a5f38a6018587d54782

Comment by Githook User [ 13/Nov/23 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-33494 Remove size storer entry upon ident drop
Branch: master
https://github.com/mongodb/mongo/commit/b4e5daaeb51920b0074661c87dc3b3765431e04e

Comment by Eric Milkie [ 10/Nov/23 ]

Does either startup-repair or the validate command audit the entries in this table? If not, should they? At least startup-repair could make sure that any orphaned entries get deleted.

Comment by Gregory Noma [ 10/Nov/23 ]

Since this ticket is focusing only on removing size storer entries upon collection drop, I filed SERVER-83120 to look into some one-time cleanup mechanism.

Comment by Eric Milkie [ 17/Oct/23 ]

Perhaps back in 2018 things worked differently when Dan G originally wrote the details for this ticket. From what Louis says, I think we should change the title and description to match today's behavior.

Comment by Louis Williams [ 17/Oct/23 ]

The size storer, as it exists currently behaves as follows:

  • When flushed periodically, it creates an entry for every collection where the key is the unique WT URI and the value is a BSON obj with the size and count information.
  • When a RecordStore is created on startup, we load the entry from the SizeStorer table into memory.
  • When we drop a collection, we do not delete the entry from the SizeStorer table or free it from the WT cache, but we delete the in-memory information in the destructor of the RecordStore.
  • On subsequent startups, the entries for each dropped collection remain in the table, but we do not load them into memory.

milkie@mongodb.com I'm not aware of where we would be leaking memory. The problem as I understand it is that we never clean up old entries in the table when we drop collections. But that memory goes away, and it never fills up space in the future, except maybe on WT pages of the size storer. Is there something I'm missing here?

Comment by Eric Milkie [ 17/Oct/23 ]

I thought we only leaked memory, not the actual records in the table that mirrors the in-memory structure? In which case, there is no garbage to collect at startup?

Comment by Steven Vannelli [ 17/Oct/23 ]
  • At a minimum, we want to have something that will garbage collect on startup. We can evaluate something that cleans it up if necessary. 
Comment by Kyle Suarez [ 02/Mar/18 ]

Is there a way to clean up the SizeStorer?

Generated at Thu Feb 08 04:33:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.