Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22793

Unbounded memory usage by long-running query using projection

    • Fully Compatible
    • ALL
    • Query 11 (03/14/16)

      • Create a collection with 240 M records containing nothing but default _id (i.e. db.c.insert({})).
      • run mongoexport with a projection stage, e.g.
        mongoexport -d test -c c --type=csv -f _id -o /dev/null
        

      Memory behavior as follows:

      • (collection is being created A-B; during this period we see WT cache usage grow, and at A we see a difference of about 500 MB between allocated memory and WT cache; this is probably a small (~3%) WT accounting error related to unaccounted for allocator overhead. This is unrelated to this ticket.)
      • starting at B the mongoexport query is running, and we see apparently linear memory growth outside WT cache ("tcmalloc allocated minus wt cache") rising to several GB
      • the growth pattern appears to correspond to a growing buffer that is doubled in size every time more space is needed; at C and D we see temporary spikes, presumably corresponding to both the old buffer and the new expanded buffer existing simultaneously while the old data is copied to the new buffer.

        1. growth.png
          104 kB
          Bruce Lucas

            Assignee:
            david.storch@mongodb.com David Storch
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: