[SERVER-16235] Performance issue in capped collections under WiredTiger Created: 19/Nov/14  Updated: 10/Sep/15  Resolved: 16/Dec/14

Status: Closed
Project: Core Server
Component/s: Performance, Storage
Affects Version/s: 2.8.0-rc0
Fix Version/s: 2.8.0-rc3

Type: Bug Priority: Critical - P2
Reporter: Bruce Lucas (Inactive) Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 1
Labels: wiredtiger
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-16775 Steady decline in performance in capp... Closed
related to SERVER-16247 Oplog declines in performance over ti... Closed
related to SERVER-16296 Producer-consumer use case shows decl... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

Seeing severe performance issue in capped collections.

  • drop-off of couple orders of magnitude in insertion rate apparently around the point where the collection size limit is reached (about 450k docs in this example).
  • insertion rate appears to continue to decline
  • compression does not matter
  • lsm is worse than record store

inserts/s after n docs:

     n     mmapv1   wt        wt        wt
                    rec str   rec str   lsm
                    snappy    no comp   snappy
 
 10000     48076    39062     39215     38314
 20000     52083    40816     39370     37735
 30000     50505    39062     39682     38461
 40000     52356    40322     38910     37878
 50000     52356    36630     37174     35971
   ...
400000     37313    37313     37174     32258
410000     38022    40485     38610     33112
420000     37037    42016     40650     35335
430000     35971    39370     39370     31347
440000     37037    38314     39525     34013
450000     38461    38314     38167     33670
460000     37593    19920     20449      3943
470000     38022     5396      6020       592
480000     38610     2217      1959       275
490000     38167     1140      1150       229
500000     37313      832       829       169

function create(size) {
    db.createCollection("c", {capped: true, size: size})
}
    
function insert(count) {
    var every = 10000
    var bulk = db.c.initializeUnorderedBulkOp();
    var t = new Date()
    for (var i=0; i<=count; i++) {
        if (i>0 && i%every==0) {
            bulk.execute();
            bulk = db.c.initializeUnorderedBulkOp();
            tt = new Date()
            print(i, Math.floor(every / (tt-t) * 1000))
            t = tt
        }
        bulk.insert({})
    }
}
 
create(10000000)
insert(500000)



 Comments   
Comment by Githook User [ 18/Dec/14 ]

Author:

{u'username': u'3rf', u'name': u'Kyle Erf', u'email': u'erf@mongodb.com'}

Message: SERVER-16235 fix dumprestore8.js on wiredtiger

Former-commit-id: 1f287c4b5f0446d54a57c482118620e1cb14ffe3
Branch: master
https://github.com/mongodb/mongo-tools/commit/a399a85683a0b9d446748a2ab819c8b13d649c5d

Comment by Githook User [ 17/Dec/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-16235: make sure to initialize capped counter in WT
Branch: master
https://github.com/mongodb/mongo/commit/77807b67089dfd7c1657051f1e674417ae8f3766

Comment by Githook User [ 17/Dec/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-16235: make repl test oplog rollover more general
Branch: master
https://github.com/mongodb/mongo/commit/9f6751d70c51cacc54a422bb6829c36738600f96

Comment by Githook User [ 17/Dec/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-16235: fix capped tests to work with WT
Branch: master
https://github.com/mongodb/mongo/commit/0c0fd893733ee8b60aed993e1ddbabac468bc89f

Comment by Githook User [ 17/Dec/14 ]

Author:

{u'username': u'3rf', u'name': u'Kyle Erf', u'email': u'erf@mongodb.com'}

Message: SERVER-16235 fix dumprestore8.js on wiredtiger

Signed-off-by: Mark Benvenuto <mark.benvenuto@mongodb.com>
Branch: master
https://github.com/mongodb/mongo/commit/6478e5ced5b924bf1306807ca087f0cbb84fe01b

Comment by Githook User [ 16/Dec/14 ]

Author:

{u'username': u'3rf', u'name': u'Kyle Erf', u'email': u'erf@mongodb.com'}

Message: SERVER-16235 fix dumprestore8.js on wiredtiger
Branch: master
https://github.com/mongodb/mongo-tools/commit/1f287c4b5f0446d54a57c482118620e1cb14ffe3

Comment by Githook User [ 16/Dec/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-16235: fix DEBUG check on WT capped collection
Branch: master
https://github.com/mongodb/mongo/commit/7917b37c07d67dfb0e4bc8336942526962dceef3

Comment by Githook User [ 16/Dec/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-16519 SERVER-16235: WT capped conflict fix and stop special casing oplog
Branch: master
https://github.com/mongodb/mongo/commit/d061b19c0bfc48a5a7e16d328919945f952c3e9f

Generated at Thu Feb 08 03:40:24 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.