Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-10395

Delete is too heavy for disk !

    • Type: Icon: Improvement Improvement
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.4.2, 2.4.5
    • Component/s: None
    • Environment:
      Redhat 5.7 , RAID6 disk , 4x 2.5GHz cpu, 8G RAM

      My mongodb keep inserting new data, at the same time, i have to remove old data in last minute, from my observation, i found that write lock of insert and remove are using the same one, how to distribute the lock? i am gonna improve my write operation. In EMC level i saw the queue of disk is 8~10, it is critical for me, the iostat when i am just insert and replication getmore as below.

      I am using 9 mongos, 1 config, 1 master, 1 secondary, 1 arbiter as my mongodb architecture

      VxVM5000 0.00 0.00 17.00 24.00 0.10 0.14 11.85 0.15 3.76 3.17 13.00
      VxVM5000 0.00 0.00 41.58 25.74 0.26 0.16 12.99 0.36 5.32 4.25 28.61
      VxVM5000 0.00 0.00 14.00 20.00 0.09 0.11 12.06 0.13 3.85 3.06 10.40
      VxVM5000 0.00 0.00 28.00 25.00 0.18 0.15 12.64 0.26 4.98 3.66 19.40
      VxVM5000 0.00 0.00 19.80 73.27 0.12 1.18 28.73 0.64 6.84 1.29 11.98
      VxVM5000 0.00 0.00 15.00 18.00 0.09 0.11 12.21 0.12 3.52 3.09 10.20
      VxVM5000 0.00 0.00 17.00 25.00 0.11 0.13 11.64 0.08 1.95 1.71 7.20
      VxVM5000 0.00 0.00 43.00 22.00 0.28 0.15 13.37 0.22 3.34 2.14 13.90
      VxVM5000 0.00 0.00 21.00 20.00 0.12 0.15 13.90 0.09 2.20 1.80 7.40

      The utilization of disk is under 30% running on sharding and replication, but when i try to remove last minute data, the utilization is gonna crazy and out of control. I observe that %util keep over 90% to 100%.
      Around 150 records insert per second, and remove the last minute data, no update operation. Amount of data 230,000,000. By the way, this situation happened in 50,000,000 records as well.

      iostat -xm 1
      VxVM5000 0.00 0.00 63.00 24.00 0.29 0.24 12.55 1.05 12.14 10.36 90.10
      VxVM5000 0.00 0.00 54.00 158.00 0.27 1.06 12.78 3.91 18.48 4.38 92.80
      VxVM5000 0.00 0.00 68.00 16.00 0.34 0.24 14.19 0.97 10.62 10.12 85.00
      VxVM5000 0.00 0.00 54.46 17.82 0.26 0.31 16.12 0.85 12.58 10.92 78.91
      VxVM5000 0.00 0.00 73.00 24.00 0.35 0.26 12.74 0.96 10.05 8.81 85.50
      VxVM5000 0.00 0.00 113.00 19.00 0.55 0.21 11.70 1.43 9.34 6.61 87.30
      VxVM5000 0.00 0.00 70.00 212.00 0.33 1.34 12.13 2.73 10.13 3.11 87.60
      VxVM5000 0.00 0.00 72.00 14.00 0.37 0.24 14.41 1.32 16.09 9.53 82.00
      VxVM5000 0.00 0.00 79.00 14.00 0.39 0.21 13.06 1.48 15.84 8.89 82.70
      VxVM5000 0.00 0.00 89.00 13.00 0.48 0.18 13.25 2.72 25.03 9.19 93.70
      VxVM5000 0.00 0.00 47.52 19.80 0.26 0.20 14.13 3.32 51.49 14.49 97.52

      Seems delete data is too heavy for mongodb, how can i solve this problem? may i manually distribute write lock? Suppose it should be run concurrency.

            Assignee:
            Unassigned Unassigned
            Reporter:
            chimengwong@gmail.com Chimeng Wong
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: