Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22793

Unbounded memory usage by long-running query using projection

    • Fully Compatible
    • ALL
    • Query 11 (03/14/16)

      • Create a collection with 240 M records containing nothing but default _id (i.e. db.c.insert({})).
      • run mongoexport with a projection stage, e.g.
        mongoexport -d test -c c --type=csv -f _id -o /dev/null

      Memory behavior as follows:

      • (collection is being created A-B; during this period we see WT cache usage grow, and at A we see a difference of about 500 MB between allocated memory and WT cache; this is probably a small (~3%) WT accounting error related to unaccounted for allocator overhead. This is unrelated to this ticket.)
      • starting at B the mongoexport query is running, and we see apparently linear memory growth outside WT cache ("tcmalloc allocated minus wt cache") rising to several GB
      • the growth pattern appears to correspond to a growing buffer that is doubled in size every time more space is needed; at C and D we see temporary spikes, presumably corresponding to both the old buffer and the new expanded buffer existing simultaneously while the old data is copied to the new buffer.

            david.storch@mongodb.com David Storch
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            0 Vote for this issue
            12 Start watching this issue