Replica set primary fragmentation/storage leak issue when repeatedly deleting+inserting

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: 8.0.10
    • Component/s: None
    • ALL
    • Hide

      docker compose up -d# Logs of running towards the replica set (this one leaks storage)docker compose logs -tf fragmentation-test-rs# Logs of running towards the standalone (this one does not leak storage)docker compose logs -tf fragmentation-test-standalone

      Show
      docker compose up -d# Logs of running towards the replica set (this one leaks storage)docker compose logs -tf fragmentation-test-rs# Logs of running towards the standalone (this one does not leak storage)docker compose logs -tf fragmentation-test-standalone
    • None
    • 3
    • TBD
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Hello,

      We've seen an issue with MongoDB replica set primaries where it is seemingly unable to reuse the disk space of documents when they are deleted and other documents inserted. Simply deleting and reinserting the same document in a loop seems to reproduce the issue, and doing a "replace_one" or "update_one" instead of "delete_one + insert_one" seems to be a workaround.

      We've not seen secondaries leak, unless they've previously been a primary with the leak, in which case they also seem to leak.

      To clarify what I mean by "leaking": Storage size for the database keeps growing, while the data size remains constant (or at least growing much slower). We've seen wild discrepancies between the two: Up to 2000x times larger storage size than data size, at which point we think MongoDB definitely should be able to reuse space instead of increasing the size of the file backing the collection yet again, but for some reason that still happens.

      I'm attaching a minimal example that both reproduces the issue, and also shows that the issue does not appear if the MongoDB is a standalone (i.e. not a replica set). Almost everything identical, except the MongoDB that will leak storage will have a replica set initialized.

      In the reproducing example this is the final output when I ran it last (10k deletes + inserts):

      1. Replica set
        data_size=0.04MiB, storage_size=196.05MiB
      2. Standalone
        data_size=0.04MiB, storage_size=0.11MiB

        1. docker-compose.yml
          2 kB
          Anders Wenhaug

            Assignee:
            Kenan Ali
            Reporter:
            Anders Wenhaug
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: