ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Cannot Reproduce
    • Priority: Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      We have a collection which, until recently, had a simple append-only usage pattern, with small documents consisting of mostly a date and a little bit of data, mostly less than 200 bytes.
      We receive about 25 of those documents per second and delete them after three months.
      There are also some large documents which have a size of a few MBs, but we receive them more sporadically.
      The collection currently has a data size of 38 GB, and normally, due to compression, has an on-disk storage size smaller than that.

      This was the usage pattern until a few weeks ago.
      Due to a bug in our data ingestion pipeline, we started writing duplicates of those large (few MB) documents every few seconds, which started to rapidly grow the size of the data and the size on disk. Since we were unable to fix the issue in the data ingestion pipeline, we instead set up a script to delete those documents every minute.

      So our usage pattern now is: about 25 small documents per second, and a large document every few seconds, which will then be deleted after less than a minute.
      The data size is now back to normal, but the storage size first stayed at the inflated level, then started to grow again, even though the data size no longer increases. It currently sits at 128 GB, for a collection of 38 GB of data.
      If copied to a new collection via $out, the storage size shrinks to a more reasonable 13 GB.

      However, despite the on-disk format being wildly inefficient, `compact` was not able to free any bytes.

      I've attached the full collection stats.

        1. stats-3.json
          22 kB
        2. image-2026-09-30-08-53-47-190.png
          image-2026-09-30-08-53-47-190.png
          42 kB

            Assignee:
            Rajesh Patil
            Reporter:
            Matthias Brehmer (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: