Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3705

mongos not closing connections, filling up disk with logs

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • 2.0.0-rc1
    • 1.8.2
    • None
    • None
    • ALL

    Description

      One of our mongos started spewing out this message:

      Mon Aug 29 00:40:36 [mongosMain] Listener: accept() returns -1 errno:24 Too many open files

      it wrote 6.5 gigabytes of identical lines (same timestamp and all), which filled up the root partition of the server. All queries running through that mongos started failing at the same time, which led to the application failing.

      However, mongos is still running, and holding 300 TCP connections open (according to lsof).

      From what I can see it started about an hour earlier with mongos not being able to connect to the cluster, which goes on until it can connect to all the nodes except one, and then just minutes before it starts spewing out the messages about too many open files it manages to connect to the last one too. Then it writes the same message until the disk runs out.

      Attachments

        Activity

          People

            Unassigned Unassigned
            iconara Theo Hultberg
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: