checkpoint took too long time

XMLWordPrintableJSON

    • Type: Question
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: 3.2.6, 3.6.3
    • Component/s: None
    • None
    • Environment:
      1.single node(no rs)
      2.16core,128G mem
      3.suse11
      4.data under /mnt/mongodb 533G
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      Our mongo runs everything OK before Sep 6th.(This time it's 3.2.6 version)

      We planed a backup operation in Sep 6th night using disk sync command. To ensure totally sync, we stop mongo and then do sync disk times. After that, we start mongo.

      Since this restart, mongo often hangs. Nearly 5~10mins one time and then everything gonna ok for 2~3mins and then again.

      Through mongostat, we found everytime when mongo going to hanging, there is a "flush".

      After some docs reading, we found checkpoint time is too long. When mongo going to checkpoint, all request hangs until it finished.

      During the checkpoint process, there is no iops in system.

      We found a similar case below:

      https://dba.stackexchange.com/questions/182542/mongodb-responds-slowly-during-the-wiredtiger-checkpoint-writing-process

      and it mentioned issue:

      https://jira.mongodb.org/browse/WT-3362

      so we update mongo to 3.6.3, things seem not that bad now. Checkpoint spend 180 second avg, and most important is during the checkpoint process, data has been cached can be read normally(when we under 3.2.6, everything stuck during the checkpoint process).But it still affect a lot.

      Can anyone help tell why restart mongo turn out such situation

      Regard

            Assignee:
            Danny Hatcher (Inactive)
            Reporter:
            hao shan
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: