[SERVER-21266] Bulk loading to micro-sharded cluster slower in 3.2.0-rc1 vs 3.0.6 Created: 03/Nov/15  Updated: 09/Dec/15  Resolved: 08/Nov/15

Status: Closed
Project: Core Server
Component/s: Write Ops
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: John Page Assignee: Misha Tyulenev
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: Sharding C (11/20/15)
Participants:

 Description   

Testing a bulk load into a sharded cluster - 40 microshards of wired tiger (aggreagtion cluster) on a 40 core server, loads of RAM, 8500 PIOPS, . Version 3.0.6 gets a steady 625,000 inserts/s - same hardware 3.2.0-rc1 only gets 520,000 inserts/s, that's a 17% performance drop.

arthur.viegers



 Comments   
Comment by John Page [ 08/Nov/15 ]

Re-tested on a more controlled environment and I now see 20% better on 3.2 as well so closing. Must have been some Amazon weirdness of PEBKAC.

Comment by Misha Tyulenev [ 05/Nov/15 ]

Cant reproduce locally, john.page could you please confirm that its reproducible and provide me with an access to the environment?

I scaled down the test to match my machine: loading 37M documents instead of 290M and running 24 shards instead of 40
I run on 24 CPU workstation (2 socket x 6 cores x 2 HT) . Used the ubuntu14.04 64bit binaries from https://www.mongodb.org/dl/linux/

Here are the results
Numbers in brackets are the thousands of inserts per second.

37390457 documents (/fastload test_result_2013.txt)

3.2 rc2 (359K)
real 1m44.820s
user 12m20.416s
sys 2m57.683s

3.2 rc1 (346K)
real 1m48.555s
user 12m25.048s
sys 2m54.465s

3.0.7 (283K)
real 2m11.883s
user 13m13.047s
sys 2m59.634s

3.0.6 rc2 (283K)
real 2m11.051s
user 13m11.384s
sys 2m57.193s

mongos 3.0.6 rc2, mongod 3.2.rc1 (306K)
real 2m2.206s
user 13m7.957s
sys 2m59.833s

Comment by Andy Schwerin [ 03/Nov/15 ]

Maybe it's an issue that appears as the number of shards increases. I think it's worth trying the 40 shards on a box similar to the one John used.

Comment by Andy Schwerin [ 03/Nov/15 ]

Misha's going to try to repro on the assumption that this is a mongos problem.

Comment by John Page [ 03/Nov/15 ]

Single config server - not sure what --configsvr does in 3.2, dont think it uses a replica

Comment by Andy Schwerin [ 03/Nov/15 ]

SCCC config servers?

Could you retest with 3.0 mongos and 3.2 mongos? I assume you'll see 3.0-ish behaviour, but want to confirm.

Comment by John Page [ 03/Nov/15 ]

I should add - Wired Tiger not MMAP

Comment by Daniel Pasette (Inactive) [ 03/Nov/15 ]
  • Is your bulk loader generating documents or something else? Can you share the script?
  • Are you loading through one or multiple mongos?
  • Do you have any log of the insertion rate or hardware stats?
  • Is this on ec2? Are you performing the comparison of 3.0 to 3.2 using the same server?
Generated at Thu Feb 08 03:56:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.