Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22793

Unbounded memory usage by long-running query using projection

    XMLWordPrintable

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Sprint:
      Query 11 (03/14/16)

      Description

      • Create a collection with 240 M records containing nothing but default _id (i.e. db.c.insert({})).
      • run mongoexport with a projection stage, e.g.

        mongoexport -d test -c c --type=csv -f _id -o /dev/null
        

      Memory behavior as follows:

      • (collection is being created A-B; during this period we see WT cache usage grow, and at A we see a difference of about 500 MB between allocated memory and WT cache; this is probably a small (~3%) WT accounting error related to unaccounted for allocator overhead. This is unrelated to this ticket.)
      • starting at B the mongoexport query is running, and we see apparently linear memory growth outside WT cache ("tcmalloc allocated minus wt cache") rising to several GB
      • the growth pattern appears to correspond to a growing buffer that is doubled in size every time more space is needed; at C and D we see temporary spikes, presumably corresponding to both the old buffer and the new expanded buffer existing simultaneously while the old data is copied to the new buffer.

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: