Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-6668

Mongos crashes consistently under high load for various reasons

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker - P1
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.0.6
    • Fix Version/s: None
    • Component/s: Sharding
    • Environment:
      Ubuntu 12.04, aws, m1.xlarge instance types
    • Operating System:
      Linux

      Description

      Mongos consistently crashes under high loads (~2500 concurrent requests). This is easily reproducible on our system. Stack traces and logs at links below.

      1) First crash stack traces, two out of three mongos died at the same time.
      https://gist.github.com/a70ccfbdd5e58430f68c

      2) mongos log: http://sprunge.us/OEiU - we noticed in this one: "ERROR: Out of file descriptors. Waiting one second before trying to accept more connections." So we increased the file descriptors and that one went away for now. Regardless, mongos should not crash like this.

      Somewhere in here, we increased the number of mongos from 3 to 12 and it seemed to work a bit better, but still easy to make it crash.

      3) And this just goes on and on, we'll restart them and easily kill them with a bit of load.

      http://sprunge.us/IfUR
      http://sprunge.us/gFSS
      http://sprunge.us/dLCj
      http://sprunge.us/XSOb
      http://sprunge.us/TZZD
      http://sprunge.us/ZNUO

        Attachments

          Activity

            People

            Assignee:
            asya Asya Kamsky
            Reporter:
            treeder Travis Reeder
            Participants:
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: