Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-7850

Config server floods log with: CMD fsync: sync:1 lock:0

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Duplicate
    • 2.2.1
    • None
    • None
    • None
    • Linux

    Description

      We have a sharded cluster of 5 mongod's. They are started like this:

      dbpath = /mnt/mongo/rs0
      shardsvr = true
      maxConns = 20000
      keyFile = keyfile
       
      # Logging
      logpath = logs/mongod-cn60.log
       
      # Replica set
      replSet = rs0
      oplogSize = 1024

      These are connected to by 96 nodes running mongos.

      There are three config servers are run like:

      dbpath = /mnt/mongo/cs0
      configsvr = true
      keyFile = keyfile
       
      # Logging
      logpath = configserver-cn60.log

      We see the config server log file flooded with:

      Wed Dec  5 18:51:44 [conn583] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn584] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn813] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn756] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn289] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn828] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn561] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn244] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn823] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn168] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn651] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn1286] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn630] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn773] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn230] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn583] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn756] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn813] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn584] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn1320] CMD fsync: sync:1 lock:0
      Wed Dec  5 18:51:44 [conn126] CMD fsync: sync:1 lock:0

      The config server has continuous 30% usage with no load on the cluster.

      It could be that each mongos is trying to balance, and this causes the config server to be permanently loaded. Disabling balancing reduces the load considerably to 5-10% CPU (cluster still not loaded). However we still see this fsync command every 30s or so from every connection.

      Is this expected? Can we enable balancing on only one mongos?

      Attachments

        Issue Links

          Activity

            People

              andre.defrere@mongodb.com Andre de Frere
              jblackburn James Blackburn
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: