[SERVER-9078] MongoDB2.4 cluster insert much more slow than 2.2. Created: 22/Mar/13 Updated: 10/Dec/14 Resolved: 12/Apr/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance |
| Affects Version/s: | 2.4.0 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | stronglee | Assignee: | James Wahlin |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Linux hz166-97 2.6.38-2-amd64 |
||
| Participants: |
| Description |
|
Hi, Please help to try to find the reason. Test schema: For higher insert speed, I pre split the collection into 60 chunks, each node contains And I found that the io load of 2.4 is higher than 2.2: iostax -m result for the test period(of one node):
2.2 cluster:
2.4 collection stats:
2.2 collection stats:
I also tested the hashed shard plan. (the record insert timestamp is used to be a shard key). The insert performance is more terrible. It took 1100 seconds to insert same records. |
| Comments |
| Comment by James Wahlin [ 12/Apr/13 ] |
|
No worries Strlee. Closing issue. |
| Comment by stronglee [ 12/Apr/13 ] |
|
Hi James, To eliminate individual difference between testing environments, I used one machine to do this test again on 2.2.4/2.4.1. I will do further tests base on a cluster as I mentioned at first, and create an other issue to report if I still get problems. Thanks, |
| Comment by James Wahlin [ 11/Apr/13 ] |
|
I performed a test between 2.2.3 and 2.4.1 where I took your example document, removed the _id field and inserted 3,000,000 times. I performed this insertion on an empty database several times and in each case 2.4.1 performed as well if not better than 2.2.3. My test was done in AWS on an m1.large instance with 4 EBS volumes in RAID 10. To help trouble shoot further, can you provide:
Also, would you be willing to share to code you are using to perform the insertions? Thanks, |
| Comment by stronglee [ 11/Apr/13 ] |
|
Hi, To simplify the test, I used a single mongod instance to do the same insert test and the result is the same. Test collection fields: A typical row of this collection: I deployed a mongod2.2.3 and a mongod2.4 on two machines with same hardware and system. And here's the test result: ver2.4 index: t_when index: t_when, op_nuid, src_role, src_item_uuid_ex, dst_role, dst_item_uuid_ex ver2.2.3 index: t_when index: t_when, op_nuid, src_role, src_item_uuid_ex, dst_role, dst_item_uuid_ex I didn't setup mms on these two test nodes. |
| Comment by James Wahlin [ 10/Apr/13 ] |
|
Let me know if you need further help on this. Thanks, |
| Comment by James Wahlin [ 02/Apr/13 ] |
|
Can you clarify your last statement? I am not sure what you mean by "more none sequential indexes" and "insert via mongos". Did you change your cluster configuration as well between your 2.2.3 and 2.4.1 test? Also, if you would be willing to share your test code and cluster setup I can try to reproduce your results. Alternately if you setup your test cluster in MMS with Munin-Node and re-ran your tests, it could help us in understanding where the bottleneck lies. Thanks, |
| Comment by stronglee [ 02/Apr/13 ] |
|
Hi James Wahlin, I am so sorry that I didn't reply to you because I was quite busy last days. BTW, I have tested the insert speed on flashcache machines and the performance is also poor in 2.4.1 than in 2.2.3. I think the key point is "more none sequential indexes" and "insert via mongos". |
| Comment by James Wahlin [ 01/Apr/13 ] |
|
Hi stronglee, Are you still encountering slowness in 2.4? If so can your provide the mongostat and top output requested by Eliot above? Thanks, |
| Comment by Eliot Horowitz (Inactive) [ 26/Mar/13 ] |
|
Can you send mongostat and top output from 2.2 and 2.4? |
| Comment by stronglee [ 23/Mar/13 ] |
|
PS again, I have set noobjcheck = true. |
| Comment by stronglee [ 23/Mar/13 ] |
|
PS. Load 330W records from a mysql table and insert them into mongodb cluster, mark down 330W means 3300,000 records here. |