Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3880

mongod abort trap after creating index "too many open files"

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.0.0
    • Component/s: Storage
    • Environment:
      uname -a
      Darwin 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
    • OS X

      After creating an index, the mongod went down. I am not sure whether the index is the problem. The db is 33GB

      mongod says:

      Thu Sep 15 19:26:27 [conn53] external sort used : 324 files in 2843 secs
      Thu Sep 15 19:26:27 [conn53] couldn't open /data/db/_tmp/esort.1316126343.1867758269//file.131 errno:24 Too many open files
      Thu Sep 15 19:26:27 [conn53] Assertion: 10308:mmap failed
      ...
      Abort trap

      and restarts

      — mongod console —

      ...
      Thu Sep 15 18:34:24 [clientcursormon] mem (MB) res:3602 virt:262933 mapped:130088
      Thu Sep 15 18:39:03 [conn53] build index <mydb>.<mycollection>

      { fieldList.field1: 1.0, fieldList.field2: 1.0 }

      ...
      19500/19715 98%
      19600/19715 99%
      19700/19715 99%
      Thu Sep 15 19:26:27 [conn53] external sort used : 324 files in 2843 secs
      Thu Sep 15 19:26:27 [conn53] couldn't open /data/db/_tmp/esort.1316126343.1867758269//file.131 errno:24 Too many open files
      Thu Sep 15 19:26:27 [conn53] Assertion: 10308:mmap failed
      0x1000898dd 0x10044f596 0x10044f928 0x100418dc8 0x100408543 0x1004091ba 0x100409ff0 0x10040b4aa 0x10040cc36 0x1003d44dc 0x1003e0f8f 0x1003e3569 0x100603394 0x100148d8c 0x10060a554 0x7fff81067fd6 0x7fff81067e89
      0 mongod 0x00000001000898dd _ZN5mongo11msgassertedEiPKc + 333
      1 mongod 0x000000010044f596 _ZN5mongo21BSONObjExternalSorter12FileIteratorC2ESs + 198
      2 mongod 0x000000010044f928 ZN5mongo21BSONObjExternalSorter8IteratorC2EPS0 + 808
      3 mongod 0x0000000100418dc8 _ZN5mongo24buildBottomUpPhases2And3INS_12BtreeData_V1EEEvbRNS_12IndexDetailsERNS_21BSONObjExternalSorterEbRSt4listINS_7DiskLocESaIS7_EEPNS_5CurOpEPNS_12SortPhaseOneERNS_19ProgressMeterHolderERNS_5TimerE + 120
      4 mongod 0x0000000100408543 _ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi + 2131
      5 mongod 0x00000001004091ba _ZN5mongo12buildAnIndexESsPNS_16NamespaceDetailsERNS_12IndexDetailsEib + 938
      6 mongod 0x0000000100409ff0 _ZN5mongo16insert_makeIndexEPNS_16NamespaceDetailsERKSsRKNS_7DiskLocE + 448
      7 mongod 0x000000010040b4aa _ZN5mongo11DataFileMgr6insertEPKcPKvibbPb + 1050
      8 mongod 0x000000010040cc36 _ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb + 70
      9 mongod 0x00000001003d44dc _ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE + 172
      10 mongod 0x00000001003e0f8f _ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE + 639
      11 mongod 0x00000001003e3569 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 4441
      12 mongod 0x0000000100603394 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 148
      13 mongod 0x0000000100148d8c _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 700
      14 mongod 0x000000010060a554 thread_proxy + 132
      15 libSystem.B.dylib 0x00007fff81067fd6 _pthread_start + 331
      16 libSystem.B.dylib 0x00007fff81067e89 thread_start + 13
      Thu Sep 15 19:26:28 terminate() called, printing stack:
      0x1005efaf9 0x7fff8a5a6ae1 0x7fff8a5a5e9c 0x7fff8a5a69fc 0x7fff81093eb1 0x7fff8109d7eb 0x10044ecec 0x10041682a 0x100408c82 0x1004091ba 0x100409ff0 0x10040b4aa 0x10040cc36 0x1003d44dc 0x1003e0f8f 0x1003e3569 0x100603394 0x100148d8c 0x10060a554 0x7fff81067fd6
      0 mongod 0x00000001005efaf9 _ZN5mongo11myterminateEv + 105
      1 libstdc++.6.dylib 0x00007fff8a5a6ae1 ZN10cxxabiv111_terminateEPFvvE + 11
      2 libstdc++.6.dylib 0x00007fff8a5a5e9c __cxa_call_terminate + 46
      3 libstdc++.6.dylib 0x00007fff8a5a69fc __gxx_personality_v0 + 1011
      4 libSystem.B.dylib 0x00007fff81093eb1 unwind_phase2 + 145
      5 libSystem.B.dylib 0x00007fff8109d7eb _Unwind_Resume + 91
      6 mongod 0x000000010044ecec _ZN5mongo21BSONObjExternalSorterD2Ev + 524
      7 mongod 0x000000010041682a _ZN5boost6detail17sp_counted_impl_pIN5mongo21BSONObjExternalSorterEE7disposeEv + 26
      8 mongod 0x0000000100408c82 _ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi + 3986
      9 mongod 0x00000001004091ba _ZN5mongo12buildAnIndexESsPNS_16NamespaceDetailsERNS_12IndexDetailsEib + 938
      10 mongod 0x0000000100409ff0 _ZN5mongo16insert_makeIndexEPNS_16NamespaceDetailsERKSsRKNS_7DiskLocE + 448
      11 mongod 0x000000010040b4aa _ZN5mongo11DataFileMgr6insertEPKcPKvibbPb + 1050
      12 mongod 0x000000010040cc36 _ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb + 70
      13 mongod 0x00000001003d44dc _ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE + 172
      14 mongod 0x00000001003e0f8f _ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE + 639
      15 mongod 0x00000001003e3569 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 4441
      16 mongod 0x0000000100603394 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 148
      17 mongod 0x0000000100148d8c _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 700
      18 mongod 0x000000010060a554 thread_proxy + 132
      19 libSystem.B.dylib 0x00007fff81067fd6 _pthread_start + 331
      Thu Sep 15 19:26:28 Got signal: 6 (Abort trap).

      Abort trap
      Ross:mongodb-osx-x86_64-2.0.0 dietz$ bin/mongod
      bin/mongod --help for help and startup options
      Thu Sep 15 19:26:31 [initandlisten] MongoDB starting : pid=1798 port=27017 dbpath=/data/db/ 64-bit host=ross.cs.umass.edu
      Thu Sep 15 19:26:31 [initandlisten] db version v2.0.0, pdfile version 4.5
      Thu Sep 15 19:26:31 [initandlisten] git version: 695c67dff0ffc361b8568a13366f027caa406222
      Thu Sep 15 19:26:31 [initandlisten] build info: Darwin erh2.10gen.cc 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_40
      Thu Sep 15 19:26:31 [initandlisten] options: {}
      Thu Sep 15 19:26:32 [initandlisten] journal dir=/data/db/journal
      Thu Sep 15 19:26:32 [initandlisten] recover begin
      Thu Sep 15 19:26:32 [initandlisten] recover lsn: 72267032
      Thu Sep 15 19:26:32 [initandlisten] recover /data/db/journal/j._58
      Thu Sep 15 19:26:32 [initandlisten] recover skipping application of section seq:71618602 < lsn:72267032
      Thu Sep 15 19:26:33 [initandlisten] recover skipping application of section seq:71677552 < lsn:72267032
      Thu Sep 15 19:26:35 [initandlisten] recover skipping application of section seq:71736532 < lsn:72267032
      Thu Sep 15 19:26:36 [initandlisten] recover skipping application of section seq:71795652 < lsn:72267032
      Thu Sep 15 19:26:38 [initandlisten] recover skipping application of section seq:71854402 < lsn:72267032
      Thu Sep 15 19:26:39 [initandlisten] recover skipping application of section seq:71912882 < lsn:72267032
      Thu Sep 15 19:26:39 [initandlisten] recover skipping application of section seq:71971892 < lsn:72267032
      Thu Sep 15 19:26:40 [initandlisten] recover /data/db/journal/j._59
      Thu Sep 15 19:26:40 [initandlisten] recover skipping application of section seq:72030732 < lsn:72267032
      Thu Sep 15 19:26:41 [initandlisten] recover skipping application of section seq:72089832 < lsn:72267032
      Thu Sep 15 19:26:42 [initandlisten] recover skipping application of section more...
      Thu Sep 15 19:26:59 [initandlisten] recover cleaning up
      Thu Sep 15 19:26:59 [initandlisten] removeJournalFiles
      Thu Sep 15 19:26:59 [initandlisten] recover done
      Thu Sep 15 19:26:59 [websvr] admin web console waiting for connections on port 28017
      Thu Sep 15 19:26:59 [initandlisten] waiting for connections on port 27017
      Thu Sep 15 19:27:59 [clientcursormon] mem (MB) res:25 virt:2426 mapped:0
      Thu Sep 15 19:32:59 [clientcursormon] mem (MB) res:26 virt:2426 mapped:0
      Thu Sep 15 19:37:59 [clientcursormon] mem (MB) res:26 virt:2426 mapped:0

      ------

      $ lsof | grep mongod
      mongod 1798 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
      mongod 1798 dietz txt REG 14,2 11817872 8867691 /opt/mongo/mongodb-osx-x86_64-2.0.0/bin/mongod
      mongod 1798 dietz txt REG 14,2 1054960 1068215 /usr/lib/dyld
      mongod 1798 dietz txt REG 14,2 233623552 8445455 /private/var/db/dyld/dyld_shared_cache_x86_64
      mongod 1798 dietz 0u CHR 16,0 0t31334702 657 /dev/ttys000
      mongod 1798 dietz 1u CHR 16,0 0t31334702 657 /dev/ttys000
      mongod 1798 dietz 2u CHR 16,0 0t31334702 657 /dev/ttys000
      mongod 1798 dietz 3r CHR 9,1 0t1023 576 /dev/urandom
      mongod 1798 dietz 4u REG 14,2 5 8732740 /data/db/mongod.lock
      mongod 1798 dietz 5w REG 14,2 8192 8884882 /data/db/journal/j._0
      mongod 1798 dietz 6u IPv4 0x1728b748 0t0 TCP *:27017 (LISTEN)
      mongod 1798 dietz 7u IPv4 0x15d946c8 0t0 TCP *:28017 (LISTEN)
      mongod 1798 dietz 8u unix 0x0f6f6a70 0t0 /tmp/mongodb-27017.sock
      lsof 1817 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
      grep 1818 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
      bash 21737 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
      bash 21819 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
      bash 23891 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
      mongo 99108 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
      mongo 99108 dietz txt REG 14,2 3870368 8867690 /opt/mongo/mongodb-osx-x86_64-2.0.0/bin/mongo

            Assignee:
            Unassigned Unassigned
            Reporter:
            dietz Laura Dietz
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: