Hello,
We've seen an issue with MongoDB replica set primaries where it is seemingly unable to reuse the disk space of documents when they are deleted and other documents inserted. Simply deleting and reinserting the same document in a loop seems to reproduce the issue, and doing a "replace_one" or "update_one" instead of "delete_one + insert_one" seems to be a workaround.
We've not seen secondaries leak, unless they've previously been a primary with the leak, in which case they also seem to leak.
To clarify what I mean by "leaking": Storage size for the database keeps growing, while the data size remains constant (or at least growing much slower). We've seen wild discrepancies between the two: Up to 2000x times larger storage size than data size, at which point we think MongoDB definitely should be able to reuse space instead of increasing the size of the file backing the collection yet again, but for some reason that still happens.
I'm attaching a minimal example that both reproduces the issue, and also shows that the issue does not appear if the MongoDB is a standalone (i.e. not a replica set). Almost everything identical, except the MongoDB that will leak storage will have a replica set initialized.
In the reproducing example this is the final output when I ran it last (10k deletes + inserts):
- Replica set
data_size=0.04MiB, storage_size=196.05MiB - Standalone
data_size=0.04MiB, storage_size=0.11MiB