Current Behavior

Compaction uses a first-fit allocation with hardcoded thresholds that determine when compaction is worthwhile:

If 20% of the file space is available in the first 80% of the file, compact the last 20%
If 10% of the file space is available in the first 90% of the file, compact the last 10%

Compact is skipped if neither of these are met.

This is a pain point for customers with large collections as a 1TB collection could have 100GB of free space available.

Proposed solution

a. Add configuration options to allow users to specify custom compaction thresholds:

compact=(threshold_percent=10)

b. Add a force mode that is generally more aggressive.