[SERVER-34604] low performance under 3.6.4 with wiredTiger storage Created: 22/Apr/18  Updated: 04/Nov/18  Resolved: 27/Sep/18

Status: Closed
Project: Core Server
Component/s: Performance
Affects Version/s: 3.6.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Adun Assignee: Kelsey Schubert
Resolution: Done Votes: 6
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-34562 I use YCSB testing comparise mongodb3... Closed
is related to SERVER-31694 17% throughput regression in insert w... Closed
is related to SERVER-36875 Compare Performance between 3.4 and 4.0 Closed
Participants:
Linked BF Score: 0

 Description   

we use YCSB to stress mongodb 3.6 vs 3.4, storage wiredTiger.

We found that 3.6 slower than 3.4 about 50%. We test 5 times, results as below, the number is qps( only insert thest)

3.6.4 | 3.4.14
------ | ------
17757 | 18622
17695 | 18841
15910 | 18265
16378 | 18504
16142 | 18480

YCSB's workload:

<noformat>
recordcount=100000000
operationcount=100000000
workload=com.yahoo.ycsb.workloads.CoreWorkload
threadcount = 50

readallfields=true

readproportion=0
updateproportion=0
scanproportion=0
insertproportion=1
readmodifywriteproportion=0

requestdistribution=zipfian
insertorder = hashed
readallfields=true
fieldlength = 200
fieldcount = 10
mongodb.writeConcern = acknowledged
mongodb.database = ycsb
mongodb.url = mongodb://192.168.1.2:27017/admin
table = ycsbtest
<noformat>

mongod's configuration as below:

<noformat>
dbpath=/home/mongo/data
syslog=true
logappend=true
bind_ip = 127.0.0.1,192.168.1.2
port = 27017

cpu = true
noauth = true

maxConns = 65535

smallfiles = true
noprealloc = true
oplogSize = 20000
wiredTigerDirectoryForIndexes = true
directoryperdb = true
wiredTigerCacheSizeGB = 16
storageEngine = wiredTiger
<noformat>

we also test:
1. replication set, but 3.6.4 is much slower too.
2. networkMessageCompressors = disabled under 3.6, no positive feedback.

1. The hardware/network/OS exactly the same(the same machine).
2. nofile/nproc both changed to 65536
3. no firewall
4. OS is Debian 9 amd64
5. SSD



 Comments   
Comment by VictorGP [ 28/Sep/18 ]

I would like to try these fixes. Will they be included in 3.6? Lot of people are stuck in 3.4 waiting for these fixes, so it would be nice if they are included in 3.6

Comment by VictorGP [ 28/Sep/18 ]

Are these fixes also related to this issue? https://jira.mongodb.org/browse/SERVER-31694

I understand you have your priorities, but a 10-20% performance degradation in a database is a serious issue. No matter how many  good features a database has, if it is not fast, and it seems MongoDB is taking the wrong route.

In SERVER-31694 it seems the performance degradation is due to bulk inserts, isn't it the most common use case for any heavy write pattern (the most common one nowadays)?

Comment by Kelsey Schubert [ 27/Sep/18 ]

Hi stutiredboy@gmail.com,

We've gone through all the related regressions in our internal performance testing. A number of fixes have gone into the server to address the issues. However, we still observe an 10-20% regression since 3.4. Consequently, we would expect that you will see some improvement if you rerun your experiment with the latest stable release, but there may still see some regression. Currently, we are prioritizing performance work in other areas and do not have additional improvements planned for this issue, so I'm going close this ticket. However, if the regression still reproduces for you and is significant, please let us know and we'll reopen the ticket.

Thank you,
Kelsey

Comment by Tomaz [ 06/Aug/18 ]

I wonder if the performance regressions I started to see running our unit tests under MongoDB 3.6 are related to this issue.

I recently upgraded mongoengine and pymongo in our project (StackStorm/st2) and updated our Python unit tests so they also run under MongoDB 3.6 on Travis (in addition to MongoDB 3.4).

Here is an example build - https://travis-ci.org/StackStorm/st2/builds/412602415 (unit tests under 3.4 run 17 minutes vs 24 minutes under 3.6).

This issue and timing can be consistently reproduced if I re-run the build.

The only thing which has changed is the database version which is used for the unit tests.

I didn't have time to really dig in yet, but since nothing else has changed, it makes it seemed that the issue is indeed related to MongoDB 3.6.3.

For reference:

Thanks,
Tomaz

 

Comment by Adun [ 21/Jun/18 ]

sorry for the typo, it is ~10%

 

Comment by Henrik Ingo (Inactive) [ 20/Jun/18 ]

Hi Adun

"We found that 3.6 slower than 3.4 about 50%."

Did you mean to say 15%? I compare the averages of the 5 results you've posted in the description, and the difference seems to be 9.5%. Am I missing some other results somewhere?

Comment by David Daly [ 02/May/18 ]

Hi Adun,

Thank for reporting this issue. Our internal automated performance testing has identified some regressions which may be related to the performance that you're observing in your tests. To most effectively work to improve performance, we're focusing on our investigation against our internal performance benchmarks which are configured to be consistent and stress particular mongod access paths. Once these issues have been resolved, we may be able to revisit this issue to see whether the same fixes resolve this particular performance regression.

Kind regards,
David

Comment by Adun [ 27/Apr/18 ]

when the insert ops become very low, the top 4 events in `perf top` are always:

   8.14%  [kernel]            [k] entry_SYSCALL_64
   7.06%  [kernel]            [k] syscall_return_via_sysret
   5.38%  [kernel]            [k] native_write_msr
   3.80%  mongod           [.] __wt_cache_eviction_worker

Comment by Adun [ 22/Apr/18 ]

YCSB's workload:

recordcount=100000000
operationcount=100000000
workload=com.yahoo.ycsb.workloads.CoreWorkload
threadcount = 50
 
readallfields=true
 
readproportion=0
updateproportion=0
scanproportion=0
insertproportion=1
readmodifywriteproportion=0
 
requestdistribution=zipfian
insertorder = hashed
readallfields=true
fieldlength = 200
fieldcount = 10
mongodb.writeConcern = acknowledged
mongodb.database = ycsb
mongodb.url = mongodb://192.168.1.2:27017/admin
table = ycsbtest

mongod's configuration as below:

dbpath=/home/mongo/data
syslog=true
logappend=true
bind_ip = 127.0.0.1,192.168.1.2
port = 27017
 
cpu = true
noauth = true
 
maxConns = 65535
 
smallfiles = true
noprealloc = true
oplogSize = 20000
wiredTigerDirectoryForIndexes = true
directoryperdb = true
wiredTigerCacheSizeGB = 16
storageEngine = wiredTiger

1. about 15% performance lost while running with standalone mode without auth
2. only performance lost while using wiredTiger Engine

Generated at Thu Feb 08 04:37:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.