[SERVER-16415] mmapv1 replica multi collection insert is 70% slower vs no replication Created: 04/Dec/14  Updated: 20/Jan/15  Resolved: 09/Jan/15

Status: Closed
Project: Core Server
Component/s: Concurrency, Replication
Affects Version/s: 2.8.0-rc1
Fix Version/s: 2.8.0-rc5

Type: Bug Priority: Major - P3
Reporter: Rui Zhang (Inactive) Assignee: Geert Bosch
Resolution: Done Votes: 0
Labels: 28qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File run.js     File run.js     File run.js    
Issue Links:
Depends
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

it seems mmapv1 insert with multiple collections does not benefit from CLL

test details

  • standalone mongod with or without journal
  • single member replica set with or without journal
  • two insert function as in the attached js file, test is done with 20 thread insert into 20 collection or single collection,
  • test with 2.8 master (6cab2a29c7dcb56ce132c200321bd1424f977f55)
  • check insert throughput from benchRun results

    r = run_multi_col();
    print(r.insert);

Observation

  • 2.6.5 and master (with single collection) have around 24-26% impact when enable replica
  • master with multiple collections show 68-72% performance impact when enable replication
  • it seems master benefit a lot from multiple collection insert in standalone mode, while replica set doesn't.

results (insert throughput)

# master with --nojournal, insert into multiple Collections
> master_noJournal_standalone
115225.80987608673
> master_noJournal_1memberReplica
32323.614478819432
> (master_noJournal_1memberReplica - master_noJournal_standalone)/ master_noJournal_standalone
-0.719475918515304
 
# master with journal, insert into multiple Collections
> master_withJournal_standalone
86472.48634513422
> master_withJournal_1memberReplica
27146.357440379936
> (master_withJournal_1memberReplica - master_withJournal_standalone)/ master_withJournal_standalone
> -0.6860694240705448
 
# 2.6.5 with --nojournal, insert into multiple Collections
> v265_noJournal_standalone = s3.insert
35971.029062602414
> v265_noJournal_1memberReplica = s2.insert
27461.29467821579
> (v265_noJournal_1memberReplica - v265_noJournal_standalone)/ v265_noJournal_standalone
-0.23657189149569932
 
# 2.6.5 with journal, insert into multiple Collections
> v265_withJournal_standalone
31201.989380142608
> v265_withJournal_1memberReplica
23050.56460661642
> (v265_withJournal_1memberReplica - v265_withJournal_standalone)/ v265_withJournal_standalone
> -0.26124695685954796
 
# master, insert into single Collections
> master_withJournal_1memberReplica_1col
19600.78767307458
> master_withJournal_standalone_1col
26011.588679734243
> (master_withJournal_1memberReplica_1col - master_withJournal_standalone_1col) / master_withJournal_standalone_1col
-0.24645941797681695



 Comments   
Comment by Githook User [ 09/Jan/15 ]

Author:

{u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}

Message: SERVER-16415: Use new OplogIntentWriteLock
Branch: master
https://github.com/mongodb/mongo/commit/49e92c75bd37f98f562603e5355fe117f35669f5

Comment by Rui Zhang (Inactive) [ 06/Jan/15 ]

schwerin here are results from windows (win2012r2 server), build with patch from MCI

windows (mmapv1)

test rc4 (with patch) rc4 2.6.6
multi-db insert (1 col per DB)      
standalone 120090 124107 116083
1 member repl 98454 38335 103215
       
multi-col insert (1 DB)      
standalone 123417 120631 20470
1 member repl 79866 38889 16461
       
single-col insert      
standalone 17109 17639 19326
1 member insert 12120 11675 15748

wiredTiger (windows, single col insert)

test rc4 with patch rc4
standalone 115595 115886
1 member repl 84100 101513
Comment by Andy Schwerin [ 05/Jan/15 ]

rui.zhang, we need numbers for 2.6.6, rc4 with and rc4 without the above patch from a Windows machine, to help us work out the details for a more general fix for this problem.

Comment by Andy Schwerin [ 22/Dec/14 ]

For legacy writes, at least, the difference between the multi-collection and single-collection cases, the performance difference comes from contention on the MODE_X lock on the oplog collection. In the single collection case, in what I believe is a representative run under vtune's concurrency analysis, we see 13.1% of time in _logOpRS, with around a third of that in the lock manager and ScopedTransaction. In the multi-collection case, it's two thirds of the _logOpRS time, which is now itself 36.3% of runtime.

repl::getNextOpTime and RCI::setMyLastOpTime have some contention in both cases, which indicates an avenue for improving the repl-vs-norepl performance.

rui.zhang, can you run the multi-collection test against 2.6.5 and 2.8.0-rc3 with the change that each collection is in its own database? New baselines for rc3 would also be nice.

Comment by Scott Hernandez (Inactive) [ 04/Dec/14 ]

Can you also do runs using write commands, as I think the current tests you are doing are for wire protocol writes. Add a "writeCmd" true field to each benchRun op to do that: https://github.com/mongodb/mongo/blob/master/jstests/noPassthroughWithMongod/benchrun_substitution.js#L6

Comment by Rui Zhang (Inactive) [ 04/Dec/14 ]

results are hard to read, put them into table (only put withJournal # here since journal does not impact the % of change)

version standalone throughput replica throughput % change
master multiple collections 86472 27146 -68.6%
master multi-col, writeCmd=true 85496 27309 -68.1%
2.6.5 multiple collections 31201 23050 -26.1%
2.6.5 multi-col writeCmd=true 30998 23028 -25.7%
2.6.5 single-col writeCmd=true 32335 23277 -28.0%
master single collection 26011 19600 -24.6%
Generated at Thu Feb 08 03:40:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.