Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16023

Open file limit causes shutdown of WT but mongod stull running

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0-rc0
    • Component/s: Storage
    • Labels:
    • Operating System:
      ALL

      Description

      Problem:

      On a long run, open file limit was reached, with this in the logs

      2014-10-15T02:47:26.388-0400 I NETWORK  [conn19835] end connection 127.0.0.1:43859 (37 connections now open)
      2014-10-15T02:47:35.350-0400 E STORAGE  WiredTiger (24) [1413355655:350274][981310047f7b8187f0000], lsm-worker: /home/ec2-user/db/collection-8096527837003006657-2064-000019.lsm: open: Too many open files
      2014-10-15T02:47:35.350-0400 E STORAGE  WiredTiger (24) [1413355655:350431][981310047f7b8187f0000], lsm-worker: Failed doing LSM switch: Too many open files
      2014-10-15T02:47:35.350-0400 E STORAGE  WiredTiger (0) [1413355655:350449][981310047f7b8187f0000], lsm-worker: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:47:35.350-0400 E STORAGE  WiredTiger (-31804) [1413355655:350472][981310047f7b8187f0000], lsm-worker: Error in LSM worker thread 3: WT_PANIC: WiredTiger library panic
      2014-10-15T02:47:35.351-0400 E STORAGE  WiredTiger (0) [1413355655:351496][98131005777b9187f0000], cursor.set_key: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:47:35.351-0400 E STORAGE  WiredTiger (0) [1413355655:351543][98131005777b9187f0000], file:WiredTiger.wt, cursor.search: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:47:35.351-0400 E STORAGE  WiredTiger (0) [1413355655:351560][98131005777b9187f0000], file:WiredTiger.wt, cursor.close: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:47:35.351-0400 E STORAGE  WiredTiger (-31804) [1413355655:351570][98131005777b9187f0000], lsm-worker: Failed doing LSM switch: WT_PANIC: WiredTiger library panic
      2014-10-15T02:47:35.351-0400 E STORAGE  WiredTiger (0) [1413355655:351580][98131005777b9187f0000], lsm-worker: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:47:35.351-0400 E STORAGE  WiredTiger (-31804) [1413355655:351596][98131005777b9187f0000], lsm-worker: Error in LSM worker thread 2: WT_PANIC: WiredTiger library panic
      2014-10-15T02:47:35.352-0400 E STORAGE  WiredTiger (0) [1413355655:352124][981310067f7b9187f0000], cursor.set_key: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:47:35.352-0400 E STORAGE  WiredTiger (0) [1413355655:352166][981310067f7b9187f0000], file:WiredTiger.wt, cursor.search: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:47:35.352-0400 E STORAGE  WiredTiger (0) [1413355655:352179][981310067f7b9187f0000], file:WiredTiger.wt, cursor.close: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:47:35.352-0400 E STORAGE  WiredTiger (-31804) [1413355655:352188][981310067f7b9187f0000], lsm-worker: Failed doing LSM switch: WT_PANIC: WiredTiger library panic
      2014-10-15T02:47:35.352-0400 E STORAGE  WiredTiger (0) [1413355655:352197][981310067f7b9187f0000], lsm-worker: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:47:35.352-0400 E STORAGE  WiredTiger (-31804) [1413355655:352206][981310067f7b9187f0000], lsm-worker: Error in LSM worker thread 1: WT_PANIC: WiredTiger library panic
      2014-10-15T02:48:08.150-0400 E STORAGE  WiredTiger (0) [1413355688:150467][98131009777bb187f0000], cursor.set_key: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:48:08.150-0400 E STORAGE  WiredTiger (0) [1413355688:150550][98131009777bb187f0000], file:WiredTiger.wt, cursor.search: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:48:08.150-0400 E STORAGE  WiredTiger (0) [1413355688:150564][98131009777bb187f0000], file:WiredTiger.wt, cursor.close: the WiredTiger library cannot continue; the process must exit and restart
      2014-10-15T02:48:08.150-0400 E STORAGE  WiredTiger (-31804) [1413355688:150575][98131009777bb187f0000], sweep-server: handle sweep server error: WT_PANIC: WiredTiger library panic
      2014-10-15T12:06:37.356-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:43896 #19872 (38 connections now open)
      2014-10-15T12:06:37.356-0400 I NETWORK  [conn19872] end connection 127.0.0.1:43896 (37 connections now open)
      2014-10-15T12:06:37.591-0400 I NETWORK  [conn3] end connection 127.0.0.1:52133 (36 connections now open)

      However, mongod left running.

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: