[SERVER-25483] 3.0.9 vs 3.2.8 Journal disk write characteristic changes. Created: 08/Aug/16  Updated: 18/Nov/16  Resolved: 18/Nov/16

Status: Closed
Project: Core Server
Component/s: MMAPv1, Performance
Affects Version/s: 3.2.8
Fix Version/s: None

Type: Question Priority: Minor - P4
Reporter: Luke Morfitt Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File disk busy time.png     PNG File disk iops.png     Text File mongomaster.txt     Text File mongoslave.txt     PNG File thoughput.png    
Issue Links:
Related
Participants:

 Description   

Hello,

We are currently reviewing the upgrade from mongo 3.0.9 to 3.2.8 in a test environment.

Our current configuration has 3x 3.0.9 nodes and 3x 3.2.8 nodes, with half of the nodes hidden at any time.

Whilst running our own application testing script against we have noticed a significant increase in the amount of data and writes to the journal disks on both the master and slaves when running 3.2.8 in comparison to 3.0.9.

We are trying to understand the changes between the versions as it may impact the performance of our applications.

Replication Process: -

Build a mongo cluster comprising of 3.2.8 nodes and 3.0.9 nodes. ( /var/lib/mongo and /var/lib/mongo/journal as dedicate separate disks)
Run a large amount of inserts (50,000 fewer may highlight the issue) whilst monitoring the dedicated journal disk.
You should see results similar the attached dataset.

Results: -
mongomaster.txt - Mongo Master running Mongo 3.0.9
mongoslave.txt - Mongo Slave running Mongo 3.28
*.png - Graph showing data gathered every minuet.

Kind regards,
Luke



 Comments   
Comment by Kelsey Schubert [ 18/Nov/16 ]

Hi lukemorfitt,

Sorry for the delay responding. It is expected that in MongoDB 3.2, journaling will do more IO. However, we do not expect to see a significant performance impact as a result of this change. It's expected that the journal will self-throttle on the secondaries if they start to lag. If you are observing a performance issue, such as slowdown in the number of database operations mongod is able to process, please open a new ticket with diagnostic.data and we will investigate.

Thank you,
Thomas

Comment by Luke Morfitt [ 08/Aug/16 ]

To rule out this being a transitional issue whilst the migration is in progress. I've removed all the 3.0.9 host from the cluster leaving a cluster of 3x 3.2.8 servers. The journal disks are showing the same increase in IOPS on the slaves. The master is showing 90 IOPS (vs 30 on 3.0.9).

Thanks
Luke

Comment by Luke Morfitt [ 08/Aug/16 ]

Please node mo03 is master(3.0.9) and mo04 is slave(3.2.8)

thanks.

Generated at Thu Feb 08 04:09:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.