[SERVER-10393] Replication is too heavy for disk Created: 01/Aug/13  Updated: 10/Dec/14  Resolved: 02/Aug/13

Status: Closed
Project: Core Server
Component/s: MapReduce, Replication
Affects Version/s: 2.4.2, 2.4.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Chimeng Wong Assignee: Unassigned
Resolution: Done Votes: 0
Labels: connection, crash, performance, replication
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Redhat 5.7 , RAID6 disk , 4x 2.5GHz cpu, 8G RAM


Operating System: Linux
Steps To Reproduce:

rs.add(" slavehost");

Participants:

 Description   

I am using 1 mongos, 1 config, 1 master, 1 secondary, 1 arbiter as my mongodb architecture, it is running on my server normal without replication, iostat as below
VxVM5000 0.00 0.00 17.00 24.00 0.10 0.14 11.85 0.15 3.76 3.17 13.00
VxVM5000 0.00 0.00 41.58 25.74 0.26 0.16 12.99 0.36 5.32 4.25 28.61
VxVM5000 0.00 0.00 14.00 20.00 0.09 0.11 12.06 0.13 3.85 3.06 10.40
VxVM5000 0.00 0.00 28.00 25.00 0.18 0.15 12.64 0.26 4.98 3.66 19.40
VxVM5000 0.00 0.00 19.80 73.27 0.12 1.18 28.73 0.64 6.84 1.29 11.98
VxVM5000 0.00 0.00 15.00 18.00 0.09 0.11 12.21 0.12 3.52 3.09 10.20
VxVM5000 0.00 0.00 17.00 25.00 0.11 0.13 11.64 0.08 1.95 1.71 7.20
VxVM5000 0.00 0.00 43.00 22.00 0.28 0.15 13.37 0.22 3.34 2.14 13.90
VxVM5000 0.00 0.00 21.00 20.00 0.12 0.15 13.90 0.09 2.20 1.80 7.40

The utilization of disk keep is under 30% running on sharding, but when i try to add the secondary, the utilization is gonna crazy and out of control. I observe that %util keep over 90%.

Around 150 records insert per second, and remove the last minute data, no update operation. Amount of data 230,000,000. By the way, this situation happened in 50,000,000 records as well.

iostat -xm 1
VxVM5000 0.00 0.00 63.00 24.00 0.29 0.24 12.55 1.05 12.14 10.36 90.10
VxVM5000 0.00 0.00 54.00 158.00 0.27 1.06 12.78 3.91 18.48 4.38 92.80
VxVM5000 0.00 0.00 68.00 16.00 0.34 0.24 14.19 0.97 10.62 10.12 85.00
VxVM5000 0.00 0.00 54.46 17.82 0.26 0.31 16.12 0.85 12.58 10.92 78.91
VxVM5000 0.00 0.00 73.00 24.00 0.35 0.26 12.74 0.96 10.05 8.81 85.50
VxVM5000 0.00 0.00 113.00 19.00 0.55 0.21 11.70 1.43 9.34 6.61 87.30
VxVM5000 0.00 0.00 70.00 212.00 0.33 1.34 12.13 2.73 10.13 3.11 87.60
VxVM5000 0.00 0.00 72.00 14.00 0.37 0.24 14.41 1.32 16.09 9.53 82.00
VxVM5000 0.00 0.00 79.00 14.00 0.39 0.21 13.06 1.48 15.84 8.89 82.70
VxVM5000 0.00 0.00 89.00 13.00 0.48 0.18 13.25 2.72 25.03 9.19 93.70
VxVM5000 0.00 0.00 47.52 19.80 0.26 0.20 14.13 3.32 51.49 14.49 97.52

Under this situation, i will get many exception of mongodb, such as bug of RS102, DR102, data too stale, out of sync , sth like that, replication is very unstable. I use it on my production already but it is unstable for me.
It is possibility occur by replication since i add secondary as slave, do i have any option to control replication getmore? any tuning option can use for stable, otherwise, i can not use it on my production.

Could you please provide some possible tuning option or experiences for me, Please help because it is so urgent for me, many thanks !



 Comments   
Comment by Daniel Pasette (Inactive) [ 02/Aug/13 ]

The SERVER project is meant for bug reporting and feature requests. Please direct questions to the mailing list: https://groups.google.com/forum/#!forum/mongodb-user

Comment by Chimeng Wong [ 01/Aug/13 ]

wrong title , please close it

Generated at Thu Feb 08 03:23:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.