[SERVER-10395] Delete is too heavy for disk ! Created: 01/Aug/13  Updated: 10/Dec/14  Resolved: 14/Aug/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.4.2, 2.4.5
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Chimeng Wong Assignee: Unassigned
Resolution: Done Votes: 0
Labels: crash, performance, replication, update
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Redhat 5.7 , RAID6 disk , 4x 2.5GHz cpu, 8G RAM


Participants:

 Description   

My mongodb keep inserting new data, at the same time, i have to remove old data in last minute, from my observation, i found that write lock of insert and remove are using the same one, how to distribute the lock? i am gonna improve my write operation. In EMC level i saw the queue of disk is 8~10, it is critical for me, the iostat when i am just insert and replication getmore as below.

I am using 9 mongos, 1 config, 1 master, 1 secondary, 1 arbiter as my mongodb architecture

VxVM5000 0.00 0.00 17.00 24.00 0.10 0.14 11.85 0.15 3.76 3.17 13.00
VxVM5000 0.00 0.00 41.58 25.74 0.26 0.16 12.99 0.36 5.32 4.25 28.61
VxVM5000 0.00 0.00 14.00 20.00 0.09 0.11 12.06 0.13 3.85 3.06 10.40
VxVM5000 0.00 0.00 28.00 25.00 0.18 0.15 12.64 0.26 4.98 3.66 19.40
VxVM5000 0.00 0.00 19.80 73.27 0.12 1.18 28.73 0.64 6.84 1.29 11.98
VxVM5000 0.00 0.00 15.00 18.00 0.09 0.11 12.21 0.12 3.52 3.09 10.20
VxVM5000 0.00 0.00 17.00 25.00 0.11 0.13 11.64 0.08 1.95 1.71 7.20
VxVM5000 0.00 0.00 43.00 22.00 0.28 0.15 13.37 0.22 3.34 2.14 13.90
VxVM5000 0.00 0.00 21.00 20.00 0.12 0.15 13.90 0.09 2.20 1.80 7.40

The utilization of disk is under 30% running on sharding and replication, but when i try to remove last minute data, the utilization is gonna crazy and out of control. I observe that %util keep over 90% to 100%.
Around 150 records insert per second, and remove the last minute data, no update operation. Amount of data 230,000,000. By the way, this situation happened in 50,000,000 records as well.

iostat -xm 1
VxVM5000 0.00 0.00 63.00 24.00 0.29 0.24 12.55 1.05 12.14 10.36 90.10
VxVM5000 0.00 0.00 54.00 158.00 0.27 1.06 12.78 3.91 18.48 4.38 92.80
VxVM5000 0.00 0.00 68.00 16.00 0.34 0.24 14.19 0.97 10.62 10.12 85.00
VxVM5000 0.00 0.00 54.46 17.82 0.26 0.31 16.12 0.85 12.58 10.92 78.91
VxVM5000 0.00 0.00 73.00 24.00 0.35 0.26 12.74 0.96 10.05 8.81 85.50
VxVM5000 0.00 0.00 113.00 19.00 0.55 0.21 11.70 1.43 9.34 6.61 87.30
VxVM5000 0.00 0.00 70.00 212.00 0.33 1.34 12.13 2.73 10.13 3.11 87.60
VxVM5000 0.00 0.00 72.00 14.00 0.37 0.24 14.41 1.32 16.09 9.53 82.00
VxVM5000 0.00 0.00 79.00 14.00 0.39 0.21 13.06 1.48 15.84 8.89 82.70
VxVM5000 0.00 0.00 89.00 13.00 0.48 0.18 13.25 2.72 25.03 9.19 93.70
VxVM5000 0.00 0.00 47.52 19.80 0.26 0.20 14.13 3.32 51.49 14.49 97.52

Seems delete data is too heavy for mongodb, how can i solve this problem? may i manually distribute write lock? Suppose it should be run concurrency.



 Comments   
Comment by Stennie Steneker (Inactive) [ 14/Aug/13 ]

Hi Chimeng,

The Core Server project is intended for filing feature requests and bug reports.

For support questions please create a new discussion in the mongodb-user group: https://groups.google.com/forum/?fromgroups#!forum/mongodb-user

Thanks,
Stephen

Generated at Thu Feb 08 03:23:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.