After creating an index, the mongod went down. I am not sure whether the index is the problem. The db is 33GB
mongod says:
Thu Sep 15 19:26:27 [conn53] external sort used : 324 files in 2843 secs
Thu Sep 15 19:26:27 [conn53] couldn't open /data/db/_tmp/esort.1316126343.1867758269//file.131 errno:24 Too many open files
Thu Sep 15 19:26:27 [conn53] Assertion: 10308:mmap failed
...
Abort trap
and restarts
— mongod console —
...
Thu Sep 15 18:34:24 [clientcursormon] mem (MB) res:3602 virt:262933 mapped:130088
Thu Sep 15 18:39:03 [conn53] build index <mydb>.<mycollection>
...
19500/19715 98%
19600/19715 99%
19700/19715 99%
Thu Sep 15 19:26:27 [conn53] external sort used : 324 files in 2843 secs
Thu Sep 15 19:26:27 [conn53] couldn't open /data/db/_tmp/esort.1316126343.1867758269//file.131 errno:24 Too many open files
Thu Sep 15 19:26:27 [conn53] Assertion: 10308:mmap failed
0x1000898dd 0x10044f596 0x10044f928 0x100418dc8 0x100408543 0x1004091ba 0x100409ff0 0x10040b4aa 0x10040cc36 0x1003d44dc 0x1003e0f8f 0x1003e3569 0x100603394 0x100148d8c 0x10060a554 0x7fff81067fd6 0x7fff81067e89
0 mongod 0x00000001000898dd _ZN5mongo11msgassertedEiPKc + 333
1 mongod 0x000000010044f596 _ZN5mongo21BSONObjExternalSorter12FileIteratorC2ESs + 198
2 mongod 0x000000010044f928 ZN5mongo21BSONObjExternalSorter8IteratorC2EPS0 + 808
3 mongod 0x0000000100418dc8 _ZN5mongo24buildBottomUpPhases2And3INS_12BtreeData_V1EEEvbRNS_12IndexDetailsERNS_21BSONObjExternalSorterEbRSt4listINS_7DiskLocESaIS7_EEPNS_5CurOpEPNS_12SortPhaseOneERNS_19ProgressMeterHolderERNS_5TimerE + 120
4 mongod 0x0000000100408543 _ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi + 2131
5 mongod 0x00000001004091ba _ZN5mongo12buildAnIndexESsPNS_16NamespaceDetailsERNS_12IndexDetailsEib + 938
6 mongod 0x0000000100409ff0 _ZN5mongo16insert_makeIndexEPNS_16NamespaceDetailsERKSsRKNS_7DiskLocE + 448
7 mongod 0x000000010040b4aa _ZN5mongo11DataFileMgr6insertEPKcPKvibbPb + 1050
8 mongod 0x000000010040cc36 _ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb + 70
9 mongod 0x00000001003d44dc _ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE + 172
10 mongod 0x00000001003e0f8f _ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE + 639
11 mongod 0x00000001003e3569 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 4441
12 mongod 0x0000000100603394 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 148
13 mongod 0x0000000100148d8c _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 700
14 mongod 0x000000010060a554 thread_proxy + 132
15 libSystem.B.dylib 0x00007fff81067fd6 _pthread_start + 331
16 libSystem.B.dylib 0x00007fff81067e89 thread_start + 13
Thu Sep 15 19:26:28 terminate() called, printing stack:
0x1005efaf9 0x7fff8a5a6ae1 0x7fff8a5a5e9c 0x7fff8a5a69fc 0x7fff81093eb1 0x7fff8109d7eb 0x10044ecec 0x10041682a 0x100408c82 0x1004091ba 0x100409ff0 0x10040b4aa 0x10040cc36 0x1003d44dc 0x1003e0f8f 0x1003e3569 0x100603394 0x100148d8c 0x10060a554 0x7fff81067fd6
0 mongod 0x00000001005efaf9 _ZN5mongo11myterminateEv + 105
1 libstdc++.6.dylib 0x00007fff8a5a6ae1 ZN10cxxabiv111_terminateEPFvvE + 11
2 libstdc++.6.dylib 0x00007fff8a5a5e9c __cxa_call_terminate + 46
3 libstdc++.6.dylib 0x00007fff8a5a69fc __gxx_personality_v0 + 1011
4 libSystem.B.dylib 0x00007fff81093eb1 unwind_phase2 + 145
5 libSystem.B.dylib 0x00007fff8109d7eb _Unwind_Resume + 91
6 mongod 0x000000010044ecec _ZN5mongo21BSONObjExternalSorterD2Ev + 524
7 mongod 0x000000010041682a _ZN5boost6detail17sp_counted_impl_pIN5mongo21BSONObjExternalSorterEE7disposeEv + 26
8 mongod 0x0000000100408c82 _ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi + 3986
9 mongod 0x00000001004091ba _ZN5mongo12buildAnIndexESsPNS_16NamespaceDetailsERNS_12IndexDetailsEib + 938
10 mongod 0x0000000100409ff0 _ZN5mongo16insert_makeIndexEPNS_16NamespaceDetailsERKSsRKNS_7DiskLocE + 448
11 mongod 0x000000010040b4aa _ZN5mongo11DataFileMgr6insertEPKcPKvibbPb + 1050
12 mongod 0x000000010040cc36 _ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb + 70
13 mongod 0x00000001003d44dc _ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE + 172
14 mongod 0x00000001003e0f8f _ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE + 639
15 mongod 0x00000001003e3569 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 4441
16 mongod 0x0000000100603394 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 148
17 mongod 0x0000000100148d8c _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 700
18 mongod 0x000000010060a554 thread_proxy + 132
19 libSystem.B.dylib 0x00007fff81067fd6 _pthread_start + 331
Thu Sep 15 19:26:28 Got signal: 6 (Abort trap).
Abort trap
Ross:mongodb-osx-x86_64-2.0.0 dietz$ bin/mongod
bin/mongod --help for help and startup options
Thu Sep 15 19:26:31 [initandlisten] MongoDB starting : pid=1798 port=27017 dbpath=/data/db/ 64-bit host=ross.cs.umass.edu
Thu Sep 15 19:26:31 [initandlisten] db version v2.0.0, pdfile version 4.5
Thu Sep 15 19:26:31 [initandlisten] git version: 695c67dff0ffc361b8568a13366f027caa406222
Thu Sep 15 19:26:31 [initandlisten] build info: Darwin erh2.10gen.cc 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_40
Thu Sep 15 19:26:31 [initandlisten] options: {}
Thu Sep 15 19:26:32 [initandlisten] journal dir=/data/db/journal
Thu Sep 15 19:26:32 [initandlisten] recover begin
Thu Sep 15 19:26:32 [initandlisten] recover lsn: 72267032
Thu Sep 15 19:26:32 [initandlisten] recover /data/db/journal/j._58
Thu Sep 15 19:26:32 [initandlisten] recover skipping application of section seq:71618602 < lsn:72267032
Thu Sep 15 19:26:33 [initandlisten] recover skipping application of section seq:71677552 < lsn:72267032
Thu Sep 15 19:26:35 [initandlisten] recover skipping application of section seq:71736532 < lsn:72267032
Thu Sep 15 19:26:36 [initandlisten] recover skipping application of section seq:71795652 < lsn:72267032
Thu Sep 15 19:26:38 [initandlisten] recover skipping application of section seq:71854402 < lsn:72267032
Thu Sep 15 19:26:39 [initandlisten] recover skipping application of section seq:71912882 < lsn:72267032
Thu Sep 15 19:26:39 [initandlisten] recover skipping application of section seq:71971892 < lsn:72267032
Thu Sep 15 19:26:40 [initandlisten] recover /data/db/journal/j._59
Thu Sep 15 19:26:40 [initandlisten] recover skipping application of section seq:72030732 < lsn:72267032
Thu Sep 15 19:26:41 [initandlisten] recover skipping application of section seq:72089832 < lsn:72267032
Thu Sep 15 19:26:42 [initandlisten] recover skipping application of section more...
Thu Sep 15 19:26:59 [initandlisten] recover cleaning up
Thu Sep 15 19:26:59 [initandlisten] removeJournalFiles
Thu Sep 15 19:26:59 [initandlisten] recover done
Thu Sep 15 19:26:59 [websvr] admin web console waiting for connections on port 28017
Thu Sep 15 19:26:59 [initandlisten] waiting for connections on port 27017
Thu Sep 15 19:27:59 [clientcursormon] mem (MB) res:25 virt:2426 mapped:0
Thu Sep 15 19:32:59 [clientcursormon] mem (MB) res:26 virt:2426 mapped:0
Thu Sep 15 19:37:59 [clientcursormon] mem (MB) res:26 virt:2426 mapped:0
------
$ lsof | grep mongod
mongod 1798 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
mongod 1798 dietz txt REG 14,2 11817872 8867691 /opt/mongo/mongodb-osx-x86_64-2.0.0/bin/mongod
mongod 1798 dietz txt REG 14,2 1054960 1068215 /usr/lib/dyld
mongod 1798 dietz txt REG 14,2 233623552 8445455 /private/var/db/dyld/dyld_shared_cache_x86_64
mongod 1798 dietz 0u CHR 16,0 0t31334702 657 /dev/ttys000
mongod 1798 dietz 1u CHR 16,0 0t31334702 657 /dev/ttys000
mongod 1798 dietz 2u CHR 16,0 0t31334702 657 /dev/ttys000
mongod 1798 dietz 3r CHR 9,1 0t1023 576 /dev/urandom
mongod 1798 dietz 4u REG 14,2 5 8732740 /data/db/mongod.lock
mongod 1798 dietz 5w REG 14,2 8192 8884882 /data/db/journal/j._0
mongod 1798 dietz 6u IPv4 0x1728b748 0t0 TCP *:27017 (LISTEN)
mongod 1798 dietz 7u IPv4 0x15d946c8 0t0 TCP *:28017 (LISTEN)
mongod 1798 dietz 8u unix 0x0f6f6a70 0t0 /tmp/mongodb-27017.sock
lsof 1817 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
grep 1818 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
bash 21737 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
bash 21819 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
bash 23891 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
mongo 99108 dietz cwd DIR 14,2 204 8867687 /opt/mongo/mongodb-osx-x86_64-2.0.0
mongo 99108 dietz txt REG 14,2 3870368 8867690 /opt/mongo/mongodb-osx-x86_64-2.0.0/bin/mongo