Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19006

mongod hang more than 30s few times a day

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.6.10
    • Fix Version/s: None
    • Component/s: MMAPv1, Replication, Usability
    • Labels:
      None
    • Operating System:
      ALL

      Description

      There is only mongod running on the machine. In daylight, mongod instance will hang more than 30s a few times.

      When mongod hanging:

      • There is no logs ouputs
      • mongostat which connected to the mongod will stop ouputing stat
      • query get no response( then timeout )
      • no disk read operations, and a few disk write operstions
      • MajorPF decrease from 20~30 to 0, MinPF decrease from 20~30k to less than 10k
      • network read traffic decrease from 600kb+ to 10~150kb, write traffic decrease from 3MB+ to 10~30kb

      Here is collected dstat data when mongod(PRIMARY) hanging:

          time     |run blk new|usr sys idl wai hiq siq| read  writ| read  writ| recv  send|  in   out | used  free|files  inodes|tot tcp udp raw frg|majpf minpf alloc  free
      17-06 10:13:05|4.0   0 1.0|  9   5  87   0   0   0|3336k 2472k|50.0  15.0 | 786k   13M|   0     0 |  38M  474M|20720  96300 |  2   2   9   0   0|  33  6634  8385  4439 
      17-06 10:13:06|3.0   0 1.0| 10   5  85   0   0   0|2728k 2328k|43.0  14.0 | 873k 9604k|   0     0 |  38M  474M|20720  96300 |  2   2   9   0   0|  23    27k   10k   12k
      17-06 10:13:07|3.0   0 7.0|  8   5  87   0   0   0|1728k 2204k|28.0  15.0 | 724k 5850k|   0     0 |  38M  474M|20720  96298 |  2   2   9   0   0|  19    31k   10k 8094 
      17-06 10:13:08|2.0   0   0|  8   5  87   0   0   0|2136k   65M|31.0  3886 | 609k 9773k|   0     0 |  38M  474M|20720  96298 |  2   2   9   0   0|  21  4047  4987  3614 
      17-06 10:13:09|2.0   0   0|  7   5  88   0   0   0|1212k 1248k|17.0  9.00 | 475k 3621k|   0     0 |  38M  474M|20720  96307 |  2   2   9   0   0|   8  7449  6747  4435 
      .... hang here .....
      17-06 10:13:10|3.0   0   0|  4   4  91   0   0   0|   0     0 |   0     0 | 121k   17k|   0     0 |  38M  474M|20720  96307 |  2   2   9   0   0|   0  1121  1095  1081 
      17-06 10:13:11|2.0   0 1.0|  4   4  91   0   0   0|   0     0 |   0     0 | 109k   19k|   0     0 |  38M  474M|20720  96307 |  2   2   9   0   0|   0  1944  1874  1875 
      17-06 10:13:12|2.0   0 7.0|  4   5  91   0   0   0|   0    16k|   0  2.00 |  95k   18k|   0     0 |  38M  474M|20720  96298 |  2   2   9   0   0|   0  8095  5435  5427 
      17-06 10:13:13|2.0   0   0|  4   4  92   0   0   0|   0     0 |   0     0 | 111k   27k|   0     0 |  38M  474M|20720  96301 |  2   2   9   0   0|   0    88    13    17 
      17-06 10:13:14|2.0   0 1.0|  4   5  91   0   0   0|   0   156k|   0  2.00 |  63k   14k|   0     0 |  38M  474M|20720  96307 |  2   2   9   0   0|   0  3235  2951  2951 
      17-06 10:13:15|2.0   0   0|  4   5  91   0   0   0|   0     0 |   0     0 |  47k   13k|   0     0 |  38M  474M|20720  96307 |  2   2   9   0   0|   0  3308  2934  2934 
      17-06 10:13:16|2.0   0 2.0|  4   4  92   0   0   0|   0     0 |   0     0 |  17k 9630B|   0     0 |  38M  474M|20720  96300 |  2   2   9   0   0|   0   268    18    18 
      17-06 10:13:17|3.0   0 5.0|  4   5  91   0   0   0|   0    16k|   0  2.00 |8419B 6844B|   0     0 |  38M  474M|20720  96301 |  2   2   9   0   0|   0  6301  4134  3144 
      17-06 10:13:18|2.0   0 3.0|  4   4  91   0   0   0|   0     0 |   0     0 |4435B 4755B|   0     0 |  38M  474M|20720  96299 |  2   2   9   0   0|   0  2101  1437  2418 
      17-06 10:13:19|2.0   0   0|  4   5  91   0   0   0|   0     0 |   0     0 |  44k   12k|   0     0 |  38M  474M|20720  96315 |  2   2   9   0   0|   0  3225  2954  2954 
      17-06 10:13:20|2.0   0   0|  4   5  91   0   0   0|   0     0 |   0     0 | 158k   29k|   0     0 |  38M  474M|20720  96315 |  2   2   9   0   0|   0  3529  2973  2973 
      17-06 10:13:21|2.0   0   0|  4   4  92   0   0   0|   0    12k|   0  2.00 | 117k   27k|   0     0 |  38M  474M|20720  96315 |  2   2   9   0   0|   0    96     2     2 
      17-06 10:13:22|2.0   0   0|  4   5  91   0   0   0|   0     0 |   0     0 | 101k   19k|   0     0 |  38M  474M|20720  96315 |  2   2   9   0   0|   0  2967  2946  2944 
      17-06 10:13:23|2.0   0 8.0|  5   5  91   0   0   0|   0    16k|   0  2.00 | 166k   28k|   0     0 |  38M  474M|20720  96315 |  2   2   9   0   0|   0  9979  6095  5909 
      17-06 10:13:24|2.0   0 1.0|  4   4  92   0   0   0|   0     0 |   0     0 |  90k   20k|   0     0 |  38M  474M|20720  96316 |  2   2   9   0   0|   0    90     8     3 
      17-06 10:13:25|2.0   0 5.0|  4   4  91   0   0   0|   0     0 |   0     0 |  52k   17k|   0     0 |  38M  474M|20720  96323 |  2   2   9   0   0|   0  4191  3261  3247 
      17-06 10:13:26|2.0   0 1.0|  4   4  92   0   0   0|   0     0 |   0     0 |  16k 8844B|   0     0 |  38M  474M|20720  96323 |  2   2   9   0   0|   0    75     6    11 
      17-06 10:13:27|2.0   0   0|  4   4  91   0   0   0|   0     0 |   0     0 |  10k 9190B|   0     0 |  38M  474M|20720  96323 |  2   2   9   0   0|   0  2986  2938  2936 
      17-06 10:13:28|2.0   0 7.0|  4   5  91   0   0   0|   0    28k|   0  4.00 |8656B 5238B|   0     0 |  38M  474M|20720  96323 |  2   2   9   0   0|   0  9222  6140  6089 
      17-06 10:13:29|2.0   0   0|  4   4  92   0   0   0|   0     0 |   0     0 |  24k 8107B|   0     0 |  38M  474M|20720  96323 |  2   2   9   0   0|   0    85     9     9 
      17-06 10:13:30|2.0   0   0|  4   4  91   0   0   0|   0     0 |   0     0 | 131k   27k|   0     0 |  38M  474M|20720  96323 |  2   2   9   0   0|   0  2994  2943  2940 
      17-06 10:13:31|2.0   0  25|  4   4  91   0   0   0|   0     0 |   0     0 | 130k   35k|   0     0 |  38M  474M|20720  96432 |  2   2   9   0   0|   0  3113  3207  3025 
      17-06 10:13:32|2.0   0  45|  4   4  92   0   0   0|   0     0 |   0     0 | 100k   34k|   0     0 |  38M  474M|20720  96478 |  2   2   9   0   0|   0   454   511    25 
      17-06 10:13:33|2.0   0  50|  5   5  91   0   0   0|   0    10M|   0   790 | 112k   34k|   0     0 |  38M  474M|20720  97036 |  2   2   9   0   0|   0    11k 7378  6799 
      17-06 10:13:34|3.0   0  62|  4   4  91   0   0   0|   0   112k|   0  8.00 | 157k   44k|   0     0 |  38M  474M|20720  97098 |  2   2   9   0   0|   0  2957  2862  2154 
      17-06 10:13:35|2.0   0  20|  4   4  92   0   0   0|   0     0 |   0     0 |  44k   15k|   0     0 |  38M  474M|20720  97719 |  2   2   9   0   0|   0  1853  1544  1321 
      17-06 10:13:36|2.0   0 9.0|  4   4  91   0   0   0|   0    52k|   0  6.00 |  20k   11k|   0     0 |  38M  474M|20720  97916 |  2   2   9   0   0|   0  3414  3362  3250 
      17-06 10:13:37|2.0   0 3.0|  4   4  92   0   0   0|   0    92k|   0  6.00 |  40k 8092B|   0     0 |  38M  474M|20720  97919 |  2   2   9   0   0|   0   647    67    21 
      17-06 10:13:38|2.0   0  11|  5   5  91   0   0   0|   0    44k|   0  2.00 |  10k   10k|   0     0 |  38M  474M|20720  97259 |  2   2   9   0   0|   0    11k 7395    22k
      17-06 10:13:39|2.0   0  32|  4   4  91   0   0   0|   0    32k|   0  6.00 |  27k   18k|   0     0 |  38M  474M|20720  97336 |  2   2   9   0   0|   0  7432  4474  4165 
      17-06 10:13:40|5.0   0 104|  4   5  91   0   0   0|   0     0 |   0     0 | 170k   63k|   0     0 |  38M  474M|20768  97441 |  2   2   9   0   0|   0  1581  1540    15k
      17-06 10:13:41|2.0   0  95|  4   5  91   0   0   0|   0     0 |   0     0 | 159k   59k|   0     0 |  38M  474M|20768  98703 |  2   2   9   0   0|   0  5230  5358    17k
      17-06 10:13:42|2.0   0  89|  4   5  91   0   0   0|   0     0 |   0     0 | 132k   54k|   0     0 |  38M  474M|20816  99730 |  3   2   9   0   0|   0  4978  5424  5233 
      17-06 10:13:43|2.0   0  83|  4   5  91   0   0   0|   0    32k|   0  4.00 | 109k   43k|   0     0 |  38M  474M|20864  99800 |  3   2   9   0   0|   0  8355  5044  5214 
      ..... resumed here ....
      17-06 10:13:44|5.0   0  78| 17   7  75   0   0   0|8052k   14M| 305  17.7k|1673k   17M|   0     0 |  38M  474M|20912  97331 |  2   2   9   0   0|  92    42k   28k   29k
      17-06 10:13:45|7.0   0   0| 20   6  74   0   0   0|2828k 2980k| 118  18.0 |1568k 6635k|   0     0 |  38M  474M|20912  97331 |  2   2   9   0   0|  32    28k   17k   12k
      17-06 10:13:46|3.0   0 2.0| 17   6  77   0   0   0|1256k 1852k|37.0  12.0 |1370k 3418k|   0     0 |  38M  474M|20912  97338 |  2   2   9   0   0|  13    21k   12k   11k
      17-06 10:13:47|6.0   0   0| 18   6  75   0   0   0|6864k 4564k| 156  22.0 |2001k   27M|   0     0 |  38M  474M|20912  97345 |  2   2   9   0   0|  58    36k   25k   23k
      
      

        Attachments

        1. mongod-gdb-bt.gz
          201 kB
        2. mongod-psaxl
          144 kB

          Activity

            People

            Assignee:
            ramon.fernandez Ramon Fernandez Marina
            Reporter:
            wujiangcheng Jiangcheng Wu
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: