[SERVER-10280] Too much database space with a queue collection Created: 22/Jul/13  Updated: 11/Jul/16  Resolved: 15/Dec/13

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.4.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: David Verdejo Assignee: Unassigned
Resolution: Done Votes: 0
Labels: database, queue, space
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

rsstuff:SECONDARY> db.serverBuildInfo()
{
"version" : "2.4.5",
"gitVersion" : "a2ddc68ba7c9cee17bfe69ed840383ec3506602b",
"sysInfo" : "windows sys.getwindowsversion(major=6, minor=1, build=7601,
platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49",
"loaderFlags" : "/nologo /LTCG /DEBUG /LARGEADDRESSAWARE /NODEFAULTLIB:M
SVCPRT",
"compilerFlags" : "/TP /nologo /EHsc /W3 /wd4355 /wd4800 /wd4267 /wd4244
/Z7 /errorReport:none /O2 /Oy- /MT /GL",
"allocator" : "system",
"versionArray" : [
2,
4,
5,
0
],
"javascriptEngine" : "V8",
"bits" : 64,
"debug" : false,
"maxBsonObjectSize" : 16777216,
"ok" : 1
}


Operating System: Windows
Steps To Reproduce:

Create a collection and insert a lot of documents and create a process to extract with findAndModify.

For example:

//server

var dormir=1000;
var umbral=1000;

function sleep(milliseconds) {
var start = new Date().getTime();
to=0;
for (var i = 0; (i < 1e7 && to==0); i++) {
if ((new Date().getTime() - start) > milliseconds)

{ to=1; }
}
}

var i=0;
while (i<umbral) {
y ="Message " + i;
db.queue.insert({_id:i,message:y});
i++;
sleep(dormir);
};

//client

var dormir=100;
var umbral=1;

function sleep(milliseconds) {
var start = new Date().getTime();
to=0;
for (var i = 0; (i < 1e7 && to==0); i++) {
if ((new Date().getTime() - start) > milliseconds){ to=1; }

}
}

var i=0;
while (i<umbral) {
y ="Message " + i;
res=db.queue.findAndModify({query:{},sort:

{"_id":1}

,remove:true});
printjson(res);
sleep(dormir);
};

Participants:

 Description   

We have a database with some collection and we use these collection like a queue. We insert documents in the queue with db.<collection>.insert() and we extract them with db.<collection>.findAndModify({query:{},sort:

{"_id":1}

,remove:true})

The problem is that we observe that the database size is bigger and bigger even when the size of the collection is small.

Now, we have the following situation:

  • Database size

rsstuff:SECONDARY> db.stats(1024*1024)
{
"db" : "colas",
"collections" : 14,
"objects" : 26529,
"avgObjSize" : 269721.5101963889,
"dataSize" : 6823,
"storageSize" : 119875,
"numExtents" : 180,
"indexes" : 13,
"indexSize" : 0,
"fileSize" : 126851,
"nsSizeMB" : 16,
"dataFileVersion" :

{ "major" : 4, "minor" : 5 }

,
"ok" : 1
}

  • Script to see size and storageSize

rsstuff:SECONDARY> escala=1024*1024
1048576
rsstuff:SECONDARY> mdb = db.getSiblingDB("colas");
colas
rsstuff:SECONDARY> mdb.getCollectionNames().forEach(function(c)

{s = mdb[c].stat s(escala); print(s.ns + " --> Count:" + s.count + "; Size(MB):" + s.size + "; Si ze in Disk(MB):" + s.storageSize)}

)
colas.Queue_FlightsTouroperationCache --> Count:0; Size(MB):0; Size in Disk(MB):
73
colas.Queue_FlightsTouroperationPurchase --> Count:0; Size(MB):0; Size in Disk(M
B):0
colas.Queue_FlightsTouroperationShowcase --> Count:0; Size(MB):0; Size in Disk(M
B):0
colas.Queue_HotelsTouroperationCache --> Count:25000; Size(MB):6823; Size in Dis
k(MB):21528
colas.Queue_HotelsTouroperationPurchase --> Count:901; Size(MB):6; Size in Disk(
MB):42004
colas.Queue_HotelsTouroperationShowcase --> Count:320; Size(MB):2; Size in Disk(
MB):28674
colas.Queue_PackagesTouroperationPRODUCT_FLIGHTHOTEL --> Count:168; Size(MB):1;
Size in Disk(MB):8965
colas.Queue_PackagesTouroperationPRODUCT_HOTEL --> Count:409; Size(MB):3; Size i
n Disk(MB):18626
colas.Queue_PackagesTouroperationProduct_Flight --> Count:0; Size(MB):0; Size in
Disk(MB):0
colas.Queue_PackagesTouroperationProduct_Flight_OW --> Count:0; Size(MB):0; Size
in Disk(MB):0
colas.Queue_PackagesTouroperationSHOWCASE --> Count:0; Size(MB):0; Size in Disk(
MB):0
colas.system.indexes --> Count:13; Size(MB):0; Size in Disk(MB):0
colas.system.users --> Count:2; Size(MB):0; Size in Disk(MB):0



 Comments   
Comment by Daniel Pasette (Inactive) [ 23/Jul/13 ]

In some workloads, mongodb can be susceptible to fragmentation and inefficient re-use of freed space in the collection. This is helped by using the powerOf2Sizes option on these collections. This can keep storage size steady in delete-heavy workloads.

See: http://docs.mongodb.org/manual/reference/command/collMod/#usePowerOf2Sizes

Generated at Thu Feb 08 03:22:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.