-
Type: Bug
-
Resolution: Unresolved
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
StorEng - Refinement Pipeline
For non-tiered storage, there is a 1-1 mapping from Btrees to Block manager to WT_BLOCK structs. For tiered storage Btrees, there is still a single Block manager (WT_BM), but that struct has an array of WT_BLOCKs that represent referenced objects. The array is built lazily, but is never shrunk. A simple sweep thread could mark and sweep WT_BLOCKs that are idle.
This not only removes unused memory, it will also make searches for needed WT_BLOCKs generally faster, and by closing the WT_BLOCK, we'll free a file descriptor when we're using dir_store. At the moment, running dir_store, we're limited to how many flush_tiers we can do because we'll use approximately (N_FLUSH_TIER * N_TABLES) file descriptors. I have a test that on Mac has about 10 tables, and after ~1000 flush tiers, we get EMFILE and some various weird behavior happens.
This limitation with dir_store was discovered while debugging WT-11512.
- depends on
-
WT-10953 Allow deletion of local files from tiered tables while table is open
- Closed