Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-10280

Too much database space with a queue collection

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.4.5
    • Component/s: Storage
    • Labels:
    • Environment:
    • Windows
    • Hide

      Create a collection and insert a lot of documents and create a process to extract with findAndModify.

      For example:

      //server

      var dormir=1000;
      var umbral=1000;

      function sleep(milliseconds) {
      var start = new Date().getTime();
      to=0;
      for (var i = 0; (i < 1e7 && to==0); i++) {
      if ((new Date().getTime() - start) > milliseconds)

      { to=1; }
      }
      }

      var i=0;
      while (i<umbral) {
      y ="Message " + i;
      db.queue.insert({_id:i,message:y});
      i++;
      sleep(dormir);
      };

      //client

      var dormir=100;
      var umbral=1;

      function sleep(milliseconds) {
      var start = new Date().getTime();
      to=0;
      for (var i = 0; (i < 1e7 && to==0); i++) {
      if ((new Date().getTime() - start) > milliseconds){ to=1; }

      }
      }

      var i=0;
      while (i<umbral) {
      y ="Message " + i;
      res=db.queue.findAndModify({query:{},sort:

      {"_id":1}

      ,remove:true});
      printjson(res);
      sleep(dormir);
      };

      Show
      Create a collection and insert a lot of documents and create a process to extract with findAndModify. For example: //server var dormir=1000; var umbral=1000; function sleep(milliseconds) { var start = new Date().getTime(); to=0; for (var i = 0; (i < 1e7 && to==0); i++) { if ((new Date().getTime() - start) > milliseconds) { to=1; } } } var i=0; while (i<umbral) { y ="Message " + i; db.queue.insert({_id:i,message:y}); i++; sleep(dormir); }; //client var dormir=100; var umbral=1; function sleep(milliseconds) { var start = new Date().getTime(); to=0; for (var i = 0; (i < 1e7 && to==0); i++) { if ((new Date().getTime() - start) > milliseconds){ to=1; } } } var i=0; while (i<umbral) { y ="Message " + i; res=db.queue.findAndModify({query:{},sort: {"_id":1} ,remove:true}); printjson(res); sleep(dormir); };

      We have a database with some collection and we use these collection like a queue. We insert documents in the queue with db.<collection>.insert() and we extract them with db.<collection>.findAndModify({query:{},sort:

      {"_id":1}

      ,remove:true})

      The problem is that we observe that the database size is bigger and bigger even when the size of the collection is small.

      Now, we have the following situation:

      • Database size

      rsstuff:SECONDARY> db.stats(1024*1024)
      {
      "db" : "colas",
      "collections" : 14,
      "objects" : 26529,
      "avgObjSize" : 269721.5101963889,
      "dataSize" : 6823,
      "storageSize" : 119875,
      "numExtents" : 180,
      "indexes" : 13,
      "indexSize" : 0,
      "fileSize" : 126851,
      "nsSizeMB" : 16,
      "dataFileVersion" :

      { "major" : 4, "minor" : 5 }

      ,
      "ok" : 1
      }

      • Script to see size and storageSize

      rsstuff:SECONDARY> escala=1024*1024
      1048576
      rsstuff:SECONDARY> mdb = db.getSiblingDB("colas");
      colas
      rsstuff:SECONDARY> mdb.getCollectionNames().forEach(function(c)

      {s = mdb[c].stat s(escala); print(s.ns + " --> Count:" + s.count + "; Size(MB):" + s.size + "; Si ze in Disk(MB):" + s.storageSize)}

      )
      colas.Queue_FlightsTouroperationCache --> Count:0; Size(MB):0; Size in Disk(MB):
      73
      colas.Queue_FlightsTouroperationPurchase --> Count:0; Size(MB):0; Size in Disk(M
      B):0
      colas.Queue_FlightsTouroperationShowcase --> Count:0; Size(MB):0; Size in Disk(M
      B):0
      colas.Queue_HotelsTouroperationCache --> Count:25000; Size(MB):6823; Size in Dis
      k(MB):21528
      colas.Queue_HotelsTouroperationPurchase --> Count:901; Size(MB):6; Size in Disk(
      MB):42004
      colas.Queue_HotelsTouroperationShowcase --> Count:320; Size(MB):2; Size in Disk(
      MB):28674
      colas.Queue_PackagesTouroperationPRODUCT_FLIGHTHOTEL --> Count:168; Size(MB):1;
      Size in Disk(MB):8965
      colas.Queue_PackagesTouroperationPRODUCT_HOTEL --> Count:409; Size(MB):3; Size i
      n Disk(MB):18626
      colas.Queue_PackagesTouroperationProduct_Flight --> Count:0; Size(MB):0; Size in
      Disk(MB):0
      colas.Queue_PackagesTouroperationProduct_Flight_OW --> Count:0; Size(MB):0; Size
      in Disk(MB):0
      colas.Queue_PackagesTouroperationSHOWCASE --> Count:0; Size(MB):0; Size in Disk(
      MB):0
      colas.system.indexes --> Count:13; Size(MB):0; Size in Disk(MB):0
      colas.system.users --> Count:2; Size(MB):0; Size in Disk(MB):0

            Assignee:
            Unassigned Unassigned
            Reporter:
            david.verdejo@logitravel.com David Verdejo
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: