Producer inserts documents, consumer uses findAndModify to find a document and remove it. Producer throttles insert rate by monitoring document count. See code below.

Performance behavior and profiling shows similarities to SERVER-16247 and SERVER-16235, even though no capped collections are involved in this case.

• Orange graph shows declining performance over time; appears to be headed to 0. Behavior appears similar to SERVER-16235 after the point where the capped collection had wrapped and deletes began occurring.
• Red graph shows that adding index on the "worker" field that is used in the findAndModify query changes the behavior but does not fix the problem. Behavior appears similar to SERVER-16247.
• Green graph shows that LSM is worse.

Profiles for both indexed and unindexed cases below. Similar to SERVER-16247 and SERVER-16235, both show a lot of time spent in __wt_btcur_next. Bar graph timeline shows correlation of time spent there with declining performance as the run progresses, confirming that the time spent in __wt_btcur_next is the reason for the performance decline.

function producer() {
    db.c.drop()
    //db.c.ensureIndex({worker:1})
    var limit = 10000
    var every = 1000
    for (var i=0; i<count; i++) {
        db.c.insert({_id:i})
        if (i>0 && i%every==0) {
            c = db.c.count()
            print('pro', i, db.c.count())
            while (c>limit) {
                sleep(100)
                c = db.c.count()
            }
        }
    }
}

function consumer(worker) {
    var every = 1000
    var t = new Date()
    for (var i=0; i<count; i++) {
        doc = db.c.findAndModify({query: {worker: null}, update: {$set: {worker: worker}}})
        if (doc) {
            db.c.remove(doc)
        }  else {
            sleep(100)
        }
        if (i>0 && i%every==0) {
            var tt = new Date()
            print('worker', worker, i, Math.floor(every/(tt-t)*1000))
            t = tt
        }
    }
}