Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-10395

Delete is too heavy for disk !

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major - P3
    • Resolution: Works as Designed
    • 2.4.2, 2.4.5
    • None
    • None
    • Redhat 5.7 , RAID6 disk , 4x 2.5GHz cpu, 8G RAM

    Description

      My mongodb keep inserting new data, at the same time, i have to remove old data in last minute, from my observation, i found that write lock of insert and remove are using the same one, how to distribute the lock? i am gonna improve my write operation. In EMC level i saw the queue of disk is 8~10, it is critical for me, the iostat when i am just insert and replication getmore as below.

      I am using 9 mongos, 1 config, 1 master, 1 secondary, 1 arbiter as my mongodb architecture

      VxVM5000 0.00 0.00 17.00 24.00 0.10 0.14 11.85 0.15 3.76 3.17 13.00
      VxVM5000 0.00 0.00 41.58 25.74 0.26 0.16 12.99 0.36 5.32 4.25 28.61
      VxVM5000 0.00 0.00 14.00 20.00 0.09 0.11 12.06 0.13 3.85 3.06 10.40
      VxVM5000 0.00 0.00 28.00 25.00 0.18 0.15 12.64 0.26 4.98 3.66 19.40
      VxVM5000 0.00 0.00 19.80 73.27 0.12 1.18 28.73 0.64 6.84 1.29 11.98
      VxVM5000 0.00 0.00 15.00 18.00 0.09 0.11 12.21 0.12 3.52 3.09 10.20
      VxVM5000 0.00 0.00 17.00 25.00 0.11 0.13 11.64 0.08 1.95 1.71 7.20
      VxVM5000 0.00 0.00 43.00 22.00 0.28 0.15 13.37 0.22 3.34 2.14 13.90
      VxVM5000 0.00 0.00 21.00 20.00 0.12 0.15 13.90 0.09 2.20 1.80 7.40

      The utilization of disk is under 30% running on sharding and replication, but when i try to remove last minute data, the utilization is gonna crazy and out of control. I observe that %util keep over 90% to 100%.
      Around 150 records insert per second, and remove the last minute data, no update operation. Amount of data 230,000,000. By the way, this situation happened in 50,000,000 records as well.

      iostat -xm 1
      VxVM5000 0.00 0.00 63.00 24.00 0.29 0.24 12.55 1.05 12.14 10.36 90.10
      VxVM5000 0.00 0.00 54.00 158.00 0.27 1.06 12.78 3.91 18.48 4.38 92.80
      VxVM5000 0.00 0.00 68.00 16.00 0.34 0.24 14.19 0.97 10.62 10.12 85.00
      VxVM5000 0.00 0.00 54.46 17.82 0.26 0.31 16.12 0.85 12.58 10.92 78.91
      VxVM5000 0.00 0.00 73.00 24.00 0.35 0.26 12.74 0.96 10.05 8.81 85.50
      VxVM5000 0.00 0.00 113.00 19.00 0.55 0.21 11.70 1.43 9.34 6.61 87.30
      VxVM5000 0.00 0.00 70.00 212.00 0.33 1.34 12.13 2.73 10.13 3.11 87.60
      VxVM5000 0.00 0.00 72.00 14.00 0.37 0.24 14.41 1.32 16.09 9.53 82.00
      VxVM5000 0.00 0.00 79.00 14.00 0.39 0.21 13.06 1.48 15.84 8.89 82.70
      VxVM5000 0.00 0.00 89.00 13.00 0.48 0.18 13.25 2.72 25.03 9.19 93.70
      VxVM5000 0.00 0.00 47.52 19.80 0.26 0.20 14.13 3.32 51.49 14.49 97.52

      Seems delete data is too heavy for mongodb, how can i solve this problem? may i manually distribute write lock? Suppose it should be run concurrency.

      Attachments

        Activity

          People

            Unassigned Unassigned
            chimengwong@gmail.com Chimeng Wong
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: