root@node:~# sudo -u mongodb mongod -vvvvv --dbpath /var/lib/mongodb/ 2014-05-25T10:03:29.328+0700 [DataFileSync] BackgroundJob starting: DataFileSync 2014-05-25T10:03:29.329+0700 shardKeyTest passed 2014-05-25T10:03:29.329+0700 isInRangeTest passed 2014-05-25T10:03:29.329+0700 shardObjTest passed 2014-05-25T10:03:29.330+0700 [initandlisten] MongoDB starting : pid=4564 port=27017 dbpath=/var/lib/mongodb/ 64-bit host=node.xyz.com 2014-05-25T10:03:29.330+0700 [initandlisten] db version v2.6.1 2014-05-25T10:03:29.331+0700 [initandlisten] git version: 4b95b086d2374bdcfcdf2249272fb552c9c726e8 2014-05-25T10:03:29.331+0700 [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49 2014-05-25T10:03:29.331+0700 [initandlisten] allocator: tcmalloc 2014-05-25T10:03:29.331+0700 [initandlisten] options: { storage: { dbPath: "/var/lib/mongodb/" }, systemLog: { verbosity: 5 } } 2014-05-25T10:03:29.368+0700 [initandlisten] flushing directory /var/lib/mongodb 2014-05-25T10:03:29.368+0700 [initandlisten] journal dir=/var/lib/mongodb/journal 2014-05-25T10:03:29.368+0700 [initandlisten] recover begin 2014-05-25T10:03:29.368+0700 [initandlisten] info no lsn file in journal/ directory 2014-05-25T10:03:29.368+0700 [initandlisten] recover lsn: 0 2014-05-25T10:03:29.368+0700 [initandlisten] recover /var/lib/mongodb/journal/j._0 2014-05-25T10:03:29.381+0700 [initandlisten] recover cleaning up 2014-05-25T10:03:29.381+0700 [initandlisten] removeJournalFiles 2014-05-25T10:03:29.597+0700 [initandlisten] flushing directory /var/lib/mongodb/journal 2014-05-25T10:03:29.627+0700 [initandlisten] removeJournalFiles end 2014-05-25T10:03:29.627+0700 [initandlisten] recover done 2014-05-25T10:03:29.627+0700 [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0 2014-05-25T10:04:04.503+0700 [initandlisten] flushing directory /var/lib/mongodb/journal 2014-05-25T10:04:04.528+0700 [initandlisten] flushing directory /var/lib/mongodb/journal 2014-05-25T10:04:04.554+0700 [initandlisten] opening db: local 2014-05-25T10:04:04.554+0700 [initandlisten] mmf open /var/lib/mongodb/local.ns 2014-05-25T10:04:04.554+0700 [initandlisten] mmf finishOpening 0x7f5c658cb000 /var/lib/mongodb/local.ns len:16777216 2014-05-25T10:04:04.554+0700 [initandlisten] mmf open /var/lib/mongodb/local.0 2014-05-25T10:04:04.554+0700 [initandlisten] mmf finishOpening 0x7f5c608cb000 /var/lib/mongodb/local.0 len:67108864 2014-05-25T10:04:04.555+0700 [initandlisten] enter repairDatabases (to check pdfile version #) 2014-05-25T10:04:04.555+0700 [initandlisten] admin 2014-05-25T10:04:04.555+0700 [initandlisten] opening db: admin 2014-05-25T10:04:04.556+0700 [initandlisten] mmf open /var/lib/mongodb/admin.ns 2014-05-25T10:04:04.556+0700 [initandlisten] mmf finishOpening 0x7f5c5b0ca000 /var/lib/mongodb/admin.ns len:16777216 2014-05-25T10:04:04.556+0700 [initandlisten] mmf open /var/lib/mongodb/admin.0 2014-05-25T10:04:04.556+0700 [initandlisten] mmf finishOpening 0x7f5c560ca000 /var/lib/mongodb/admin.0 len:67108864 2014-05-25T10:04:04.558+0700 [initandlisten] mmf close /var/lib/mongodb/admin.0 2014-05-25T10:04:04.558+0700 [initandlisten] _groupCommit 2014-05-25T10:04:04.558+0700 [initandlisten] journal REMAPPRIVATEVIEW 2014-05-25T10:04:04.558+0700 [initandlisten] journal REMAPPRIVATEVIEW done startedAt: 0 n:4 0ms 2014-05-25T10:04:04.558+0700 [initandlisten] groupCommit end 2014-05-25T10:04:04.558+0700 [initandlisten] mmf close /var/lib/mongodb/admin.ns 2014-05-25T10:04:04.558+0700 [initandlisten] _groupCommit 2014-05-25T10:04:04.558+0700 [initandlisten] journal REMAPPRIVATEVIEW 2014-05-25T10:04:04.558+0700 [initandlisten] journal REMAPPRIVATEVIEW done startedAt: 0 n:1 0ms 2014-05-25T10:04:04.558+0700 [initandlisten] groupCommit end 2014-05-25T10:04:04.559+0700 [initandlisten] local 2014-05-25T10:04:04.559+0700 [initandlisten] mmf close /var/lib/mongodb/local.0 2014-05-25T10:04:04.559+0700 [initandlisten] _groupCommit 2014-05-25T10:04:04.559+0700 [initandlisten] journal REMAPPRIVATEVIEW 2014-05-25T10:04:04.559+0700 [initandlisten] journal REMAPPRIVATEVIEW done startedAt: 1 n:1 0ms 2014-05-25T10:04:04.559+0700 [initandlisten] groupCommit end 2014-05-25T10:04:04.559+0700 [initandlisten] mmf close /var/lib/mongodb/local.ns 2014-05-25T10:04:04.559+0700 [initandlisten] _groupCommit 2014-05-25T10:04:04.559+0700 [initandlisten] journal REMAPPRIVATEVIEW 2014-05-25T10:04:04.559+0700 [initandlisten] journal REMAPPRIVATEVIEW done startedAt: 0 n:1 0ms 2014-05-25T10:04:04.559+0700 [initandlisten] groupCommit end 2014-05-25T10:04:04.559+0700 [initandlisten] vorchat 2014-05-25T10:04:04.559+0700 [initandlisten] opening db: vorchat 2014-05-25T10:04:04.559+0700 [initandlisten] mmf open /var/lib/mongodb/vorchat.ns 2014-05-25T10:04:04.559+0700 [initandlisten] mmf finishOpening 0x7f5c658cb000 /var/lib/mongodb/vorchat.ns len:16777216 2014-05-25T10:04:04.559+0700 [initandlisten] mmf open /var/lib/mongodb/vorchat.0 2014-05-25T10:04:04.559+0700 [initandlisten] mmf finishOpening 0x7f5c608cb000 /var/lib/mongodb/vorchat.0 len:67108864 2014-05-25T10:04:04.560+0700 [initandlisten] mmf close /var/lib/mongodb/vorchat.0 2014-05-25T10:04:04.560+0700 [initandlisten] _groupCommit 2014-05-25T10:04:04.560+0700 [initandlisten] journal REMAPPRIVATEVIEW 2014-05-25T10:04:04.560+0700 [initandlisten] journal REMAPPRIVATEVIEW done startedAt: 0 n:1 0ms 2014-05-25T10:04:04.560+0700 [initandlisten] groupCommit end 2014-05-25T10:04:04.560+0700 [initandlisten] mmf close /var/lib/mongodb/vorchat.ns 2014-05-25T10:04:04.560+0700 [initandlisten] _groupCommit 2014-05-25T10:04:04.560+0700 [initandlisten] journal REMAPPRIVATEVIEW 2014-05-25T10:04:04.560+0700 [initandlisten] journal REMAPPRIVATEVIEW done startedAt: 1 n:1 0ms 2014-05-25T10:04:04.560+0700 [initandlisten] groupCommit end 2014-05-25T10:04:04.560+0700 [initandlisten] done repairDatabases 2014-05-25T10:04:04.560+0700 [initandlisten] opening db: admin 2014-05-25T10:04:04.560+0700 [initandlisten] mmf open /var/lib/mongodb/admin.ns 2014-05-25T10:04:04.561+0700 [initandlisten] mmf finishOpening 0x7f5c658cb000 /var/lib/mongodb/admin.ns len:16777216 2014-05-25T10:04:04.561+0700 [initandlisten] mmf open /var/lib/mongodb/admin.0 2014-05-25T10:04:04.561+0700 [initandlisten] mmf finishOpening 0x7f5c608cb000 /var/lib/mongodb/admin.0 len:67108864 2014-05-25T10:04:04.561+0700 [initandlisten] Running query: query: {} sort: {} projection: {} 2014-05-25T10:04:04.561+0700 [initandlisten] query admin.system.roles planSummary: EOF ntoreturn:0 ntoskip:0 keyUpdates:0 numYields:0 locks(micros) W:577 r:268 nreturned:0 reslen:20 0ms 2014-05-25T10:04:04.561+0700 [snapshot] BackgroundJob starting: snapshot 2014-05-25T10:04:04.561+0700 [ClientCursorMonitor] BackgroundJob starting: ClientCursorMonitor 2014-05-25T10:04:04.562+0700 [initandlisten] Using idhack: query: { _id: "authSchema" } sort: {} projection: {} 2014-05-25T10:04:04.561+0700 [TTLMonitor] BackgroundJob starting: TTLMonitor 2014-05-25T10:04:04.561+0700 [PeriodicTaskRunner] BackgroundJob starting: PeriodicTaskRunner 2014-05-25T10:04:04.562+0700 [initandlisten] fd limit hard:65000 soft:62000 max conn: 49600 2014-05-25T10:04:04.562+0700 [IndexRebuilder] BackgroundJob starting: IndexRebuilder 2014-05-25T10:04:04.566+0700 [initandlisten] run command local.$cmd { create: "startup_log", size: 10485760, capped: true } 2014-05-25T10:04:04.567+0700 [initandlisten] opening db: local 2014-05-25T10:04:04.567+0700 [initandlisten] mmf open /var/lib/mongodb/local.ns 2014-05-25T10:04:04.567+0700 [initandlisten] mmf finishOpening 0x7f5c580c4000 /var/lib/mongodb/local.ns len:16777216 2014-05-25T10:04:04.567+0700 [initandlisten] mmf open /var/lib/mongodb/local.0 2014-05-25T10:04:04.567+0700 [initandlisten] mmf finishOpening 0x7f5c530c4000 /var/lib/mongodb/local.0 len:67108864 2014-05-25T10:04:04.567+0700 [initandlisten] create collection local.startup_log { size: 10485760, capped: true } 2014-05-25T10:04:04.567+0700 [initandlisten] command local.$cmd command: create { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 numYields:0 reslen:75 0ms 2014-05-25T10:04:04.567+0700 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching 2014-05-25T10:04:04.568+0700 [initandlisten] insert local.startup_log ninserted:1 keyUpdates:0 numYields:0 0ms 2014-05-25T10:04:04.568+0700 [initandlisten] waiting for connections on port 27017 2014-05-25T10:04:04.572+0700 [IndexRebuilder] opening db: vorchat 2014-05-25T10:04:04.572+0700 [IndexRebuilder] mmf open /var/lib/mongodb/vorchat.ns 2014-05-25T10:04:04.573+0700 [IndexRebuilder] mmf finishOpening 0x7f5c4e0c4000 /var/lib/mongodb/vorchat.ns len:16777216 2014-05-25T10:04:04.573+0700 [IndexRebuilder] mmf open /var/lib/mongodb/vorchat.0 2014-05-25T10:04:04.573+0700 [IndexRebuilder] mmf finishOpening 0x7f5c490c4000 /var/lib/mongodb/vorchat.0 len:67108864 2014-05-25T10:04:04.579+0700 [IndexRebuilder] IndexRebuilder::checkNS: admin.system.indexes 2014-05-25T10:04:04.579+0700 [IndexRebuilder] IndexRebuilder::checkNS: admin.system.namespaces 2014-05-25T10:04:04.579+0700 [IndexRebuilder] IndexRebuilder::checkNS: admin.system.version 2014-05-25T10:04:04.580+0700 [IndexRebuilder] IndexRebuilder::checkNS: 41f51478ea6e6f7 2014-05-25T10:04:04.580+0700 [IndexRebuilder] opening db: 41f51478ea6e6f7 2014-05-25T10:04:04.580+0700 [IndexRebuilder] IndexRebuilder::checkNS: en 2014-05-25T10:04:04.580+0700 [IndexRebuilder] opening db: en 2014-05-25T10:04:04.580+0700 [IndexRebuilder] IndexRebuilder::checkNS: ea4fd02be5900686d9f 2014-05-25T10:04:04.580+0700 [IndexRebuilder] opening db: ea4fd02be5900686d9f 2014-05-25T10:04:04.580+0700 [IndexRebuilder] IndexRebuilder::checkNS: a83 2014-05-25T10:04:04.580+0700 [IndexRebuilder] opening db: a83 2014-05-25T10:04:04.580+0700 [IndexRebuilder] IndexRebuilder::checkNS: 304a1871697549efebb7280 2014-05-25T10:04:04.580+0700 [IndexRebuilder] opening db: 304a1871697549efebb7280 2014-05-25T10:04:04.580+0700 [IndexRebuilder] IndexRebuilder::checkNS: 7e22964 2014-05-25T10:04:04.580+0700 [IndexRebuilder] opening db: 7e22964 2014-05-25T10:04:04.580+0700 [IndexRebuilder] IndexRebuilder::checkNS: 3f8884be16e1e272197b56b3830 2014-05-25T10:04:04.580+0700 [IndexRebuilder] opening db: 3f8884be16e1e272197b56b3830 2014-05-25T10:04:04.581+0700 [IndexRebuilder] IndexRebuilder::checkNS: ae8a20aad3c 2014-05-25T10:04:04.581+0700 [IndexRebuilder] opening db: ae8a20aad3c 2014-05-25T10:04:04.581+0700 [IndexRebuilder] IndexRebuilder::checkNS: 615d0f3f8af82141fdf704bdd354202 2014-05-25T10:04:04.581+0700 [IndexRebuilder] opening db: 615d0f3f8af82141fdf704bdd354202 2014-05-25T10:04:04.581+0700 [IndexRebuilder] IndexRebuilder::checkNS: 6e6025a8ba2cd57 2014-05-25T10:04:04.581+0700 [IndexRebuilder] opening db: 6e6025a8ba2cd57 2014-05-25T10:04:04.581+0700 [IndexRebuilder] IndexRebuilder::checkNS: en 2014-05-25T10:04:04.581+0700 [IndexRebuilder] IndexRebuilder::checkNS: 8111b8026c549b4c26b 2014-05-25T10:04:04.581+0700 [IndexRebuilder] opening db: 8111b8026c549b4c26b 2014-05-25T10:04:04.581+0700 [IndexRebuilder] IndexRebuilder::checkNS: 2014-05-25T10:04:04.581+0700 [IndexRebuilder] Fatal Assertion 16253 2014-05-25T10:04:04.589+0700 [IndexRebuilder] 0x11c0e91 0x1163109 0x114576d 0xa17c16 0xa17cf3 0x8effb0 0xb84753 0xb85628 0x1148922 0x1205829 0x7f5c69979e9a 0x7f5c68c8c3fd mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11c0e91] mongod(_ZN5mongo10logContextEPKc+0x159) [0x1163109] mongod(_ZN5mongo13fassertFailedEi+0xcd) [0x114576d] mongod(_ZN5mongo4Lock7DBWrite6lockDBERKSs+0x2c6) [0xa17c16] mongod(_ZN5mongo4Lock7DBWriteC1ERKNS_10StringDataE+0x63) [0xa17cf3] mongod(_ZN5mongo6Client12WriteContextC1ERKSsS3_+0x40) [0x8effb0] mongod(_ZN5mongo14IndexRebuilder7checkNSERKSt4listISsSaISsEE+0xd3) [0xb84753] mongod(_ZN5mongo14IndexRebuilder3runEv+0x188) [0xb85628] mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0xd2) [0x1148922] mongod() [0x1205829] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f5c69979e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5c68c8c3fd] 2014-05-25T10:04:04.589+0700 [IndexRebuilder] ***aborting after fassert() failure 2014-05-25T10:04:04.596+0700 [IndexRebuilder] SEVERE: Got signal: 6 (Aborted). Backtrace:0x11c0e91 0x11c026e 0x7f5c68bce4a0 0x7f5c68bce425 0x7f5c68bd1b8b 0x11457da 0xa17c16 0xa17cf3 0x8effb0 0xb84753 0xb85628 0x1148922 0x1205829 0x7f5c69979e9a 0x7f5c68c8c3fd mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11c0e91] mongod() [0x11c026e] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f5c68bce4a0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f5c68bce425] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f5c68bd1b8b] mongod(_ZN5mongo13fassertFailedEi+0x13a) [0x11457da] mongod(_ZN5mongo4Lock7DBWrite6lockDBERKSs+0x2c6) [0xa17c16] mongod(_ZN5mongo4Lock7DBWriteC1ERKNS_10StringDataE+0x63) [0xa17cf3] mongod(_ZN5mongo6Client12WriteContextC1ERKSsS3_+0x40) [0x8effb0] mongod(_ZN5mongo14IndexRebuilder7checkNSERKSt4listISsSaISsEE+0xd3) [0xb84753] mongod(_ZN5mongo14IndexRebuilder3runEv+0x188) [0xb85628] mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0xd2) [0x1148922] mongod() [0x1205829] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f5c69979e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5c68c8c3fd]