Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-47650

mongostat queue qrw 600+ arw fixed 128

    • Type: Icon: Bug Bug
    • Resolution: Community Answered
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.0.16
    • Component/s: Diagnostics
    • Labels:
      None
    • Fully Compatible
    • ALL

      mongo cluster has two shard using hash policy.i test on mongo 4.0.16.

       

      mongostat on shard 1 show qrw+arw less queue.but mongostat on shard arw fixed 128 queue,qarw 600+,mongotop for shard collection,400-600s write,but in mongo log,sql execute time is 100-500ms.so my cluster ops is 5000/s,my machine is 56c +512g +12tb ssd 10 with sas interface.

      mongo cluster:

      p  s a (shard1)

      a  s  p(shard2)

      one machine:shard 1 primary + shard2 a

      one mchine:shard 1 second + shard2 second

      one machine:shard2 primary + shard 1 a

      [shard1]

      insert query update delete getmore command dirty used flushes vsize res qrw arw net_in net_out conn set repl time
       1298 1296 1295 *0 497 598|0 4.0% 80.0% 0 235G 224G 0|1 1|0 5.23m 5.04m 70 myShard_1 PRI Apr 18 17:42:01.888
       1306 1308 1309 *0 499 598|0 4.0% 80.0% 0 235G 224G 0|0 1|0 5.28m 5.02m 70 myShard_1 PRI Apr 18 17:42:02.888
       1264 1263 1262 *0 522 617|0 4.0% 80.0% 0 235G 224G 0|0 1|0 5.15m 4.92m 70 myShard_1 PRI Apr 18 17:42:03.887
       1383 1382 1382 *0 507 614|0 4.1% 80.0% 0 235G 224G 0|0 1|0 5.56m 5.29m 70 myShard_1 PRI Apr 18 17:42:04.887
       1316 1316 1318 *0 517 623|0 4.1% 80.0% 0 235G 224G 0|0 2|0 5.33m 5.09m 70 myShard_1 PRI Apr 18 17:42:05.887
      

      [shard2]

      insert query update delete getmore command dirty used flushes vsize res qrw arw net_in net_out conn set repl time
       1169 1188 1189 *0 0 7|0 15.5% 68.4% 0 242G 206G 0|665 5|128 4.18m 2.28m 926 myShard_2 PRI Apr 18 17:29:24.155
       1075 1074 1072 *0 0 4|0 15.5% 68.4% 0 242G 206G 0|658 3|128 3.84m 2.09m 926 myShard_2 PRI Apr 18 17:29:25.153
       1115 1111 1112 *0 0 6|0 15.5% 68.4% 0 242G 206G 0|666 3|128 3.92m 2.18m 926 myShard_2 PRI Apr 18 17:29:26.155
       1198 1207 1206 *0 0 5|0 15.5% 68.4% 0 242G 206G 0|662 4|128 4.31m 2.33m 925 myShard_2 PRI Apr 18 17:29:27.153
       1182 1173 1173 *0 0 2|0 15.5% 68.4% 0 242G 206G 0|661 5|128 4.15m 2.26m 926 myShard_2 PRI Apr 18 17:29:28.153
       1212 1219 1218 *0 0 7|0 15.5% 68.4% 0 242G 206G 0|662 4|128 4.34m 2.35m 926 myShard_2 PRI Apr 18 17:29:29.154
       1174 1190 1188 *0 0 5|0 15.5% 68.4% 0 242G 207G 0|658 5|128 4.23m 2.29m 926 myShard_2 PRI Apr 18 17:29:30.154
       1247 1252 1251 *0 0 13|0 15.5% 68.4% 0 242G 207G 0|652 5|128 4.43m 2.45m 926 myShard_2 PRI Apr 18 17:29:31.155
       1234 1229 1221 *0 0 45|0 15.5% 68.4% 0 242G 207G 0|659 5|128 4.35m 2.58m 926 myShard_2 PRI Apr 18 17:29:32.154
       1214 1221 1221 *0 0 24|0 15.5% 68.4% 0 242G 207G 0|660 5|128 4.31m 2.42m 926 myShard_2 PRI Apr 18 17:29:33.154
      insert query update delete getmore command dirty used flushes vsize res qrw arw net_in net_out conn set repl time
       1236 1225 1224 *0 0 15|0 15.5% 68.4% 0 242G 207G 0|657 5|128 4.36m 2.37m 920 myShard_2 PRI Apr 18 17:29:34.154
       1197 1186 1188 *0 0 13|0 15.5% 68.4% 0 242G 207G 0|661 5|128 4.25m 2.31m 920 myShard_2 PRI Apr 18 17:29:35.154
       1259 1297 1296 *0 0 16|0 15.5% 68.4% 0 242G 207G 0|657 6|128 4.55m 2.54m 920 myShard_2 PRI Apr 18 17:29:36.155
      

        1. mongstat.jpg
          mongstat.jpg
          237 kB
        2. mongtop.jpg
          mongtop.jpg
          93 kB
        3. myShard_3.png
          myShard_3.png
          133 kB
        4. nopswithmoredirty.jpg
          nopswithmoredirty.jpg
          216 kB
        5. ops.jpg
          ops.jpg
          55 kB
        6. quque1.jpg
          quque1.jpg
          73 kB
        7. readconcern.png
          readconcern.png
          40 kB
        8. shard1mongstat.jpg
          shard1mongstat.jpg
          227 kB
        9. shard2mongo.tar.gz
          47.85 MB
        10. shard2topandstat.jpg
          shard2topandstat.jpg
          114 kB
        11. shardops.jpg
          shardops.jpg
          72 kB
        12. shardshowlog.jpg
          shardshowlog.jpg
          37 kB
        13. srvdb1.yto.cloud.png
          srvdb1.yto.cloud.png
          139 kB
        14. upload ftdc.png
          upload ftdc.png
          8 kB

            Assignee:
            dmitry.agranat@mongodb.com Dmitry Agranat
            Reporter:
            601290552@qq.com jing xu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: