[SERVER-2430] Indexed capped collection performance degrades after there are thousands of entries Created: 28/Jan/11  Updated: 29/May/12  Resolved: 02/Sep/11

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.6.3
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Joseph Wang Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File MongoMortgageQueryOffersCache.java    
Participants:

 Description   

Initially, two capped collections were created to implement "cache" feature.

db.createCollection("purchase",

{capped:true, size:10000000}

)
db.purchase.ensureIndex(

{key:1, ts:-1}

,

{background:true}

);
db.createCollection("refinance",

{capped:true, size:10000000}

)
db.refinance.ensureIndex(

{key:1, ts: -1}

,

{background:true}

);

The lookup query is done via

Date now = new Date();
BasicDBObject dbQuery = new BasicDBObject();
dbQuery.put(KEY_FIELD_NAME, requestSignature);
dbQuery.put(TIMESTAMP_FIELD_NAME, new BasicDBObject("$gt",
now.getTime() - query_offer_cache_expiration));

cur = coll.find(dbQuery).addOption(
Bytes.QUERYOPTION_SLAVEOK).sort(
getMongoDBSortCriteria()).limit(1);

We've three servers, each has 24 CPUs w/ total 48G.

We see the query performance seems to degrade after we've thousands of documents. To get thing moving again, we dropped capped collections and recreated them. Now stats whos

> db.refinance.stats()
{
"ns" : "mortgagecache.refinance",
"count" : 591,
"size" : 1235880,
"avgObjSize" : 2091.167512690355,
"storageSize" : 10000128,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 10000128,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 65536,
"indexSizes" :

{ "key_1_ts_-1" : 65536 }

,
"capped" : 1,
"max" : 2147483647,
"ok" : 1
}

Attaching the cache code



 Comments   
Comment by Eliot Horowitz (Inactive) [ 02/Sep/11 ]

Note this may have been addressed in 1.8 with better index compaction

Comment by Eliot Horowitz (Inactive) [ 06/Mar/11 ]

Can you provide stats when things are not performing well?
For example, if your index is bigger than ram and your'e doing random reads, it will be slow.

Comment by Joseph Wang [ 04/Mar/11 ]

any comment? suggestion?

Comment by Joseph Wang [ 02/Feb/11 ]

for capped (indexed) collection, what is the performance difference between capping the number of entries vs the size?

Generated at Thu Feb 08 02:59:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.