I am using 1 mongos, 1 config, 1 master, 1 secondary, 1 arbiter as my mongodb architecture, it is running on my server normal without replication, iostat as below
VxVM5000 0.00 0.00 17.00 24.00 0.10 0.14 11.85 0.15 3.76 3.17 13.00
VxVM5000 0.00 0.00 41.58 25.74 0.26 0.16 12.99 0.36 5.32 4.25 28.61
VxVM5000 0.00 0.00 14.00 20.00 0.09 0.11 12.06 0.13 3.85 3.06 10.40
VxVM5000 0.00 0.00 28.00 25.00 0.18 0.15 12.64 0.26 4.98 3.66 19.40
VxVM5000 0.00 0.00 19.80 73.27 0.12 1.18 28.73 0.64 6.84 1.29 11.98
VxVM5000 0.00 0.00 15.00 18.00 0.09 0.11 12.21 0.12 3.52 3.09 10.20
VxVM5000 0.00 0.00 17.00 25.00 0.11 0.13 11.64 0.08 1.95 1.71 7.20
VxVM5000 0.00 0.00 43.00 22.00 0.28 0.15 13.37 0.22 3.34 2.14 13.90
VxVM5000 0.00 0.00 21.00 20.00 0.12 0.15 13.90 0.09 2.20 1.80 7.40

The utilization of disk keep is under 30% running on sharding, but when i try to add the secondary, the utilization is gonna crazy and out of control. I observe that %util keep over 90%.

Around 150 records insert per second, and remove the last minute data, no update operation. Amount of data 230,000,000. By the way, this situation happened in 50,000,000 records as well.

iostat -xm 1
VxVM5000 0.00 0.00 63.00 24.00 0.29 0.24 12.55 1.05 12.14 10.36 90.10
VxVM5000 0.00 0.00 54.00 158.00 0.27 1.06 12.78 3.91 18.48 4.38 92.80
VxVM5000 0.00 0.00 68.00 16.00 0.34 0.24 14.19 0.97 10.62 10.12 85.00
VxVM5000 0.00 0.00 54.46 17.82 0.26 0.31 16.12 0.85 12.58 10.92 78.91
VxVM5000 0.00 0.00 73.00 24.00 0.35 0.26 12.74 0.96 10.05 8.81 85.50
VxVM5000 0.00 0.00 113.00 19.00 0.55 0.21 11.70 1.43 9.34 6.61 87.30
VxVM5000 0.00 0.00 70.00 212.00 0.33 1.34 12.13 2.73 10.13 3.11 87.60
VxVM5000 0.00 0.00 72.00 14.00 0.37 0.24 14.41 1.32 16.09 9.53 82.00
VxVM5000 0.00 0.00 79.00 14.00 0.39 0.21 13.06 1.48 15.84 8.89 82.70
VxVM5000 0.00 0.00 89.00 13.00 0.48 0.18 13.25 2.72 25.03 9.19 93.70
VxVM5000 0.00 0.00 47.52 19.80 0.26 0.20 14.13 3.32 51.49 14.49 97.52

Under this situation, i will get many exception of mongodb, such as bug of RS102, DR102, data too stale, out of sync , sth like that, replication is very unstable. I use it on my production already but it is unstable for me.
It is possibility occur by replication since i add secondary as slave, do i have any option to control replication getmore? any tuning option can use for stable, otherwise, i can not use it on my production.

Could you please provide some possible tuning option or experiences for me, Please help because it is so urgent for me, many thanks !