Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22793

Unbounded memory usage by long-running query using projection

    XMLWordPrintable

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Sprint:
      Query 11 (03/14/16)

      Description

      • Create a collection with 240 M records containing nothing but default _id (i.e. db.c.insert({})).
      • run mongoexport with a projection stage, e.g.

        mongoexport -d test -c c --type=csv -f _id -o /dev/null
        

      Memory behavior as follows:

      • (collection is being created A-B; during this period we see WT cache usage grow, and at A we see a difference of about 500 MB between allocated memory and WT cache; this is probably a small (~3%) WT accounting error related to unaccounted for allocator overhead. This is unrelated to this ticket.)
      • starting at B the mongoexport query is running, and we see apparently linear memory growth outside WT cache ("tcmalloc allocated minus wt cache") rising to several GB
      • the growth pattern appears to correspond to a growing buffer that is doubled in size every time more space is needed; at C and D we see temporary spikes, presumably corresponding to both the old buffer and the new expanded buffer existing simultaneously while the old data is copied to the new buffer.

        Attachments

          Activity

            People

            Assignee:
            david.storch David Storch
            Reporter:
            bruce.lucas Bruce Lucas
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: