Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22752

WiredTiger reach max 20010 sessions: Cannot allocate memory

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 3.0.8, 3.0.9
    • Component/s: WiredTiger
    • Labels:
      None
    • ALL

      Hi All,
      Today around 1 AM (GMT +2) the primary mongo crashed and other 3 mongo instances out of 5 we have in our replica set, failed successively. We are using mongodb-org-server-3.0.8-1.amzn1.x86_64 with wiredTiger engine

      The reason: It indicates we were hitting the 20k session limit, we understand that it might be due to not properly closing cursors but it seems to happen too often and cause the primary to shutdown instead of reject the new sessions.

      I start all the servers again and this is the number I had after the 4th mongo was up,

      db.serverStatus().wiredTiger.session
      { "open cursor count" : 124618, "open session count" : 17609 }
      

      When I tried to load the 5th one it failed for exceeding the limit of 20010 sessions

      It was obvious that there were too many open sessions, in order to close them manually. I decided to set the timeout to 5Sec by running

      db.runCommand({setParameter:1, cursorTimeoutMillis: 5000})
      { "was" : 600000, "ok" : 1 }
      

      It seems that the default was 600 sec! After doing it and 5 sec later I run the command again and found that number of sessions decreased to 357

      { "open cursor count" : 15585, "open session count" : 357 }
      

      I upload the last mongo and everything looks fine now.

      It is same behavior as described by Glen Miner at SERVER-17364

      I attached the all information of our server (output of mdiag.sh) and snippet of the log from time of the crash.

      meanwhile I set the default cursor time our to 30 sec just to be on the safe side even though 5 sec seems OK as well.

        1. mdiag-ip-10-100-1-154.txt.gz
          0.1 kB
        2. mongolog.txt
          27 kB

            Assignee:
            Unassigned Unassigned
            Reporter:
            dror@amobee.com Dror Bar-Gil
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: