[SERVER-20303] Negative scaling at low thread count under WiredTiger when inserting large documents Created: 05/Sep/15  Updated: 11/Jan/16  Resolved: 10/Oct/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.6, 3.1.7
Fix Version/s: 3.0.8, 3.2.0-rc0

Type: Bug Priority: Critical - P2
Reporter: Bruce Lucas (Inactive) Assignee: Keith Bostic (Inactive)
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File 3.0.0.png     PNG File 3.0.6.png     PNG File 3.1.0.png     PNG File 3.1.7.png     HTML File gdbmon.html     Text File gdbmon.log     PNG File gdbmon.png     Text File iostat.log     PNG File latest.png     Text File ss.log     HTML File stats.html     PNG File stats.png    
Issue Links:
Depends
is depended on by SERVER-21442 WiredTiger changes for MongoDB 3.0.8 Closed
is depended on by WT-1973 MongoDB changes for WiredTiger 2.7.0 Closed
Related
related to SERVER-20409 Negative scaling with more than 10K c... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Participants:

 Description   
  • test inserts 100 kB documents into a single initially empty collection from multiple client threads
  • 24 cpus (12 cores * 2 hyperthreads), 64 GB memory, SSD storage
  • standalone mongod (so no oplog); no journal; checkpoints disabled to reduce variability
  • mongod restarted between each test for repeatability

Following graph shows 5 successive runs (with mongod restart between each run) at 1, 2, 4, 8 and 16 client threads:

  • aggregate throughput declines as number of client threads increases
  • disk is mostly idle and becomes more so as op rate declines
  • cpu is mostly idle at all thread counts
  • eviction activity increases as client threads increase
  • yet bytes in cache also grows as thread count increases - eviction is somehow unable to keep up with the (increasingly meager) insertion rate?
  • memory allocations increase dramatically in spite of smaller insertion rate
  • threads appear to be blocked trying to acquire pages, probably accounting for lower insertion rate

Following gdbmon profile showing two consecutive runs with 2 and 16 threads respectively:

In the second half of the timeline, showing the run with 16 client threads,

  • A - much of the time many or most of the 16 threads are waiting (sleeping)
  • B - while one thread is evicting


 Comments   
Comment by Daniel Pasette (Inactive) [ 10/Oct/15 ]

Verified this fix after the latest WT code drop on a 24 core box.

Before:
1 thread : 9319
2 threads : 11434
4 threads : 9766
8 threadss : 4188
16 threads : 898

After:
1 threads : 13566
2 threads : 24115
4 threads : 36321
8 threads : 45561
16 threads : 45304

Comment by Githook User [ 06/Oct/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: SERVER-20303 Require a minimum item count for in-memory splits.

Merge pull request #2236 from wiredtiger/SERVER-20303-mjc

(cherry picked from commit f12d478229822ee19b5767e05e73083adbe095f4)
Branch: mongodb-3.0
https://github.com/wiredtiger/wiredtiger/commit/01dbcf110a971b8155ad76ff3b2c8b528c1d9b73

Comment by Githook User [ 06/Oct/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: SERVER-20303 Require a minimum item count for in-memory splits.

Merge pull request #2236 from wiredtiger/SERVER-20303-mjc

(cherry picked from commit f12d478229822ee19b5767e05e73083adbe095f4)
Branch: mongodb-3.0
https://github.com/wiredtiger/wiredtiger/commit/01dbcf110a971b8155ad76ff3b2c8b528c1d9b73

Comment by Githook User [ 01/Oct/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: Merge pull request #2233 from wiredtiger/SERVER-20303

SERVER-20303: tuning in-memory splits for when inserting large objects
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/ad56c6a7f9256fd0377d6441425a64df6760e6be

Comment by Githook User [ 01/Oct/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: Merge pull request #2233 from wiredtiger/SERVER-20303

SERVER-20303: tuning in-memory splits for when inserting large objects
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/ad56c6a7f9256fd0377d6441425a64df6760e6be

Comment by Githook User [ 01/Oct/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: Merge pull request #2236 from wiredtiger/SERVER-20303-mjc

SERVER-20303 Require a minimum item count for in-memory splits.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/f12d478229822ee19b5767e05e73083adbe095f4

Comment by Githook User [ 01/Oct/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: Merge pull request #2236 from wiredtiger/SERVER-20303-mjc

SERVER-20303 Require a minimum item count for in-memory splits.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/f12d478229822ee19b5767e05e73083adbe095f4

Comment by Githook User [ 01/Oct/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: SERVER-20303 Require a minimum item count for in-memory splits.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/4123e425cbcf67ee4910ec778655b9218bc82ac7

Comment by Githook User [ 01/Oct/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

Message: SERVER-20303: This change tunes for a test with a maximum page size of
10MB and multi-threaded append of 100K key/value pairs. That means 100
inserts is sufficient to trigger forced eviction, and the previous test
would refuse in-memory splits unless there were approximately 4K items
on the insert list, that is, it assumed small insert objects. Change
the code to allow in-memory splits for both large numbers of items and
large insert objects.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/a2349b6a03a68fa914c8c9acabae592da0302955

Comment by Alexander Gorrod [ 29/Sep/15 ]

> an in-memory split only ever results in 2 pages, we just create a new page and move the tail of the insert list to it.

I always forget that - I tangle up "split" with reconciliation split. In that case I see even less reason to worry about the number of items on a page.

Comment by Bruce Lucas (Inactive) [ 06/Sep/15 ]

Here's the repro script:

db=/ssd/db
 
counts="1 2 4 8 16"
seconds=30
 
killall -9 -w mongod
iostat -t -x 1 -k >iostat.log &
rm -f ss.log
 
for threads in $counts; do
    echo === $threads threads
    killall -9 -w mongod
    rm -rf $db
    mkdir -p $db
    mongod --dbpath $db --logpath $db.log --logappend --oplogSize 1000000 --fork \
        --storageEngine wiredTiger --nojournal --syncdelay 0 
    mongo --eval "while(true) {print(JSON.stringify(db.serverStatus({tcmalloc:1}))); sleep(1000)}" >>ss.log &
    mongo --eval "
        x = ''
        for (var i=0; i<100000; i++)
            x += 'x'
        printjson(benchRun({
            ops: [{
                op: 'insert',
                ns: 'test.c',
                doc: {x:x},
                //safe: true,
            }],
            seconds: $seconds,
            parallel: $threads
        }))
    "
done

Did a quick comparison of several versions, saw little difference:

3.0.0

3.0.6

3.1.0

3.1.7

latest

Generated at Thu Feb 08 03:53:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.