[SERVER-16919] Oplog can grow much larger than configured size under WiredTiger Created: 17/Jan/15  Updated: 27/Apr/15  Resolved: 22/Jan/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 2.8.0-rc5
Fix Version/s: 3.0.0-rc6

Type: Bug Priority: Major - P3
Reporter: Bruce Lucas (Inactive) Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: wiredtiger
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-16868 Significant slowdown in capped collec... Closed
is duplicated by SERVER-16868 Significant slowdown in capped collec... Closed
Related
is related to SERVER-17033 Improve performance for bulk insert i... Closed
Tested
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

Under heavy insert or update load the oplog can grow much larger than the configured size, possibly indefinitely. I've triggered this with a few different workloads, sometimes reproducibly and sometimes not. 100 threads of the following with a configured oplog size of 50 MB in a 1-node replica set seems to do it fairly reliably:

function repro(t) {
 
    db.c.drop()
    db.c.insert({_id:t, i:0})
 
    big = ''
    for (var i=0; i<100000; i++)
        big += 'x'
 
    count = 1000000
    every = 100
    for (var i=0; i<count; ) {
        var bulk = db.c.initializeOrderedBulkOp();
        for (var j=0; j<every; j++, i++)
            bulk.insert({x:big})
        try {
            bulk.execute();
        } catch(e) {
            print(t, 'OOPS')
        }
        if (t==1 && i%100==0)
            print('MB', db.getSiblingDB('local').oplog.rs.stats(1024*1024).size)
    }
}

Oplog growth as reported by the above code for two different runs, in both cases going well past the configured 50 MB size:

first run:
MB 421
MB 335
MB 1381
MB 2260

second run:
MB 263
MB 989
MB 1387
MB 2417

In both cases the cache size was 5GB, but virtual memory grew to about 8GB, and mongod was killed by the OOM killer.

I think the issue is that there's no back-pressure in capped collection inserts to throttle inserts to prevent inserts from outrunning deletes.



 Comments   
Comment by Githook User [ 22/Jan/15 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-16919 Improve WT capped collection age-out

Generated at Thu Feb 08 03:42:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.