Uploaded image for project: 'Node.js Driver'
  1. Node.js Driver
  2. NODE-507

Repeatedly Switching DBs on a single connection increases memory usage

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.0.40
    • Affects Version/s: 2.0.37
    • Component/s: None
    • Environment:
      Ubuntu 12.04, Node 0.10.38

      I came across this issue in production, after migrating from v1.4.

      Using the old driver we would create a connection to a server, then repeatedly switch databases on the same server.

      e.g. I have a list of items, each goes to a collection in a different DB:

      Pseudocode
      var db;
      // Initial Connect Using MongoClient, keep a reference to the original DB
      MongoClient.connect(url, options, function(err, initialDB){
      db = initialDB;
      // For each item, change the DB and collection
          db = db.db(dbNameForThisItem);
          var collection = db.collection(collectionNameForThisItem);
          // And insert the item
          collection.insert(item);
      

      This works fine for version 1.4.x, but in 2.0.x the memory usage increases over time (and fairly rapidly).

      I've created a repo with a test app, which switches DB in this way, dumping out the heap usage and RSS (and deltas from a baseline) every 10k insertions and dumps the heap at the end.

      https://github.com/dhendo/node-mongodb-memorytest

      To use:

      make v1

      to run against v1.4.37

      make v2

      to run against v2.0.39

      Sample output from v2:

      50.00k Collections
      201.09MB Heap
      225.48MB RSS
      ------------------
      177.56MB Heap Delta
      3.55MB Heap / 1000 collections
      ------------------
      193.36MB RSS Delta
      3.87MB RSS / 1000 collections
      ------------------
      { rss: 236437504, heapTotal: 210854144, heapUsed: 166121040 }
      

      v1:

      50.00k Collections
      29.69MB Heap
      52.50MB RSS
      ------------------
      5.09MB Heap Delta
      0.10MB Heap / 1000 collections
      ------------------
      20.89MB RSS Delta
      0.42MB RSS / 1000 collections
      ------------------
      { rss: 55046144, heapTotal: 31127296, heapUsed: 10873576 }
      

      It appears that this is due to the way that DB.db() now works - each call to it adds a reference to the "parentDb", and returns a new db object. This results in a long chain of parent/child relationships when .db() is used in this way. There isn't a leak as such, as the parentDb reference keeps all of them in scope.

      Possible Options:

      • Change the way the referencing works, so that the new parentDb is always set as the "root" db e.g.
        options.parentDb = this.parentDb || this;

        This will probably only flatten out the structure, not simplify it.

      • Introduce some level of caching on the db instances (and introduce an option to return a non-cached instance.
      • Kill the parent/child link so the GC can clean up unused db instances
      • Document this behaviour of Db.db() with a warning so that heavy users can code around it

        1. v1.4.9.png
          v1.4.9.png
          26 kB
        2. v2.0.37.png
          v2.0.37.png
          10 kB
        3. vLatest.png
          vLatest.png
          12 kB

            Assignee:
            christkv Christian Amor Kvalheim
            Reporter:
            david.henderson@triggeredmessaging.com David Henderson
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: