|
Hi hq9000,
Sorry for the delay. From your description, the corruption likely was result of the hard-reboot of the system running the VM. Unfortunately, there is little MongoDB can do if the storage layer beneath it cannot maintain guarantees about its writes during power failure.
Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.
Kind regards,
Kelsey
|
|
the same has just happened to me running 3.2.15 on centos 7 on virtualbox on my laptop.
it's MMAP database previously used with 3.0.something, which a while ago I upgraged to 3.2.15 (without touching dbpath, only binary upgrade).
the laptop had frozen and had to be hard-rebooted (powed off) with VM running inside, after which this problem has started.
It's not a production db, of course, but still looks kind of scary.
mongod log when it tries to start:
2017-08-08T13:50:55.871+0300 I CONTROL [main] ***** SERVER RESTARTED *****
|
2017-08-08T13:50:55.899+0300 I CONTROL [initandlisten] MongoDB starting : pid=2954 port=27017 dbpath=/storage/mongo 64-bit host=localhost.localdomain
|
2017-08-08T13:50:55.899+0300 I CONTROL [initandlisten] db version v3.2.15
|
2017-08-08T13:50:55.899+0300 I CONTROL [initandlisten] git version: e11e3c1b9c9ce3f7b4a79493e16f5e4504e01140
|
2017-08-08T13:50:55.900+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
|
2017-08-08T13:50:55.900+0300 I CONTROL [initandlisten] allocator: tcmalloc
|
2017-08-08T13:50:55.900+0300 I CONTROL [initandlisten] modules: none
|
2017-08-08T13:50:55.900+0300 I CONTROL [initandlisten] build environment:
|
2017-08-08T13:50:55.900+0300 I CONTROL [initandlisten] distmod: rhel70
|
2017-08-08T13:50:55.901+0300 I CONTROL [initandlisten] distarch: x86_64
|
2017-08-08T13:50:55.902+0300 I CONTROL [initandlisten] target_arch: x86_64
|
2017-08-08T13:50:55.902+0300 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "disabled", javascriptEnabled: false }, storage: { dbPath: "/storage/mongo", mmapv1: { smallFiles: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
|
2017-08-08T13:50:55.986+0300 I - [initandlisten] Detected data files in /storage/mongo created by the 'mmapv1' storage engine, so setting the active storage engine to 'mmapv1'.
|
2017-08-08T13:50:55.986+0300 W - [initandlisten] Detected unclean shutdown - /storage/mongo/mongod.lock is not empty.
|
2017-08-08T13:50:55.998+0300 I STORAGE [initandlisten]
|
2017-08-08T13:50:55.999+0300 I STORAGE [initandlisten] ** WARNING: Readahead for /storage/mongo is set to 4096KB
|
2017-08-08T13:50:55.999+0300 I STORAGE [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
|
2017-08-08T13:50:55.999+0300 I STORAGE [initandlisten] ** http://dochub.mongodb.org/core/readahead
|
2017-08-08T13:50:56.000+0300 I JOURNAL [initandlisten] journal dir=/storage/mongo/journal
|
2017-08-08T13:50:56.001+0300 I JOURNAL [initandlisten] recover : no journal files present, no recovery needed
|
2017-08-08T13:50:56.002+0300 I JOURNAL [initandlisten] preallocating a journal file /storage/mongo/journal/prealloc.0
|
2017-08-08T13:50:56.870+0300 I JOURNAL [durability] Durability thread started
|
2017-08-08T13:50:56.872+0300 I JOURNAL [journal writer] Journal writer thread started
|
2017-08-08T13:50:56.876+0300 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
|
2017-08-08T13:50:56.876+0300 I CONTROL [initandlisten]
|
2017-08-08T13:50:56.876+0300 I CONTROL [initandlisten]
|
2017-08-08T13:50:56.877+0300 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
|
2017-08-08T13:50:56.877+0300 I CONTROL [initandlisten] ** We suggest setting it to 'never'
|
2017-08-08T13:50:56.877+0300 I CONTROL [initandlisten]
|
2017-08-08T13:50:56.878+0300 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
|
2017-08-08T13:50:56.878+0300 I CONTROL [initandlisten] ** We suggest setting it to 'never'
|
2017-08-08T13:50:56.879+0300 I CONTROL [initandlisten]
|
2017-08-08T13:50:58.154+0300 I - [initandlisten] Invariant failure _name == nsToDatabaseSubstring(ns) src/mongo/db/catalog/database.cpp 410
|
2017-08-08T13:50:58.154+0300 I - [initandlisten]
|
|
***aborting after invariant() failure
|
|
|
2017-08-08T13:50:58.180+0300 F - [initandlisten] Got signal: 6 (Aborted).
|
|
0x134aea2 0x1349ff9 0x134a802 0x7fd053a26100 0x7fd05368b5f7 0x7fd05368cce8 0x12cb0fb 0xb0b062 0xb0b1d8 0xb0cd74 0xb10fec 0x9be50a 0x9c114f 0x97709d 0x7fd053677b15 0x9bc597
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"F4AEA2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F49FF9"},{"b":"400000","o":"F4A802"},{"b":"7FD053A17000","o":"F100"},{"b":"7FD053656000","o":"355F7","s":"gsignal"},{"b":"7FD053656000","o":"36CE8","s":"abort"},{"b":"400000","o":"ECB0FB","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"70B062","s":"_ZNK5mongo8Database13getCollectionENS_10StringDataE"},{"b":"400000","o":"70B1D8","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"70CD74","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"710FEC","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},{"b":"400000","o":"5BE50A"},{"b":"400000","o":"5C114F"},{"b":"400000","o":"57709D","s":"main"},{"b":"7FD053656000","o":"21B15","s":"__libc_start_main"},{"b":"400000","o":"5BC597"}],"processInfo":{ "mongodbVersion" : "3.2.15", "gitVersion" : "e11e3c1b9c9ce3f7b4a79493e16f5e4504e01140", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.3.1.el7.x86_64", "version" : "#1 SMP Wed Dec 9 14:09:15 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "FF944F0DB846B8A41B6AE667A07DD5B371773034" }, { "b" : "7FFD5BABE000", "elfType" : 3, "buildId" : "6D522BD99C31E96B22FABF33083A3E17E0A09D79" }, { "b" : "7FD05493E000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7E24873B9C50842F0D3285B7D06F872226D17262" }, { "b" : "7FD054557000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "9523142ED92FE40B988E6EED854D79BF587954C5" }, { "b" : "7FD05434F000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "CB0D2C9F29DBD13C47E7D2EEFB94B35835698CCA" }, { "b" : "7FD05414B000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "091060A163E7EDA25572F3B1BAF2E8F80209C00E" }, { "b" : "7FD053E49000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F9DF294FB70243549DCB643F1322BB20E70E9FE8" }, { "b" : "7FD053C33000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6AA1DCC4DE7F1836344949857FC2017278631FFD" }, { "b" : "7FD053A17000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "723F0AC75EF88E778940AE8A8BC30141D85B116A" }, { "b" : "7FD053656000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "2434E3F90543A7D58D480E8B8C4FAFACEA7FF1D6" }, { "b" : "7FD054BAB000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "09E1BB4D034C7263810A41100647068858A7ECB6" }, { "b" : "7FD05340A000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "D46A230FFF4A7B808B3CFC213D31FCAC542FB504" }, { "b" : "7FD053125000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6D6136A0E795420B05854DEF13A10C226FE9CCB2" }, { "b" : "7FD052F21000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "3A1166709F88740C49E060731832E3FAD2DFB66B" }, { "b" : "7FD052CEF000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "AA97A848DD7C9E57B06EC913E10D420AEBBCE027" }, { "b" : "7FD052AD9000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "1982C8CDAE90F898D1AD26DC07E807333B4789D0" }, { "b" : "7FD0528CA000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "AEF6C3D3C5152F339942041519A106FC055DAF71" }, { "b" : "7FD0526C6000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7FD0524AC000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "A9DA5486FACCE0E1DED291E6B4638CCC25F77BA0" }, { "b" : "7FD052287000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, { "b" : "7FD052026000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "30FA397B01197ECABC647CBD8E75FDD5B743D730" }, { "b" : "7FD051E01000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "98131C9354279ABD39FD80D4BE5B3EC5678BD9E0" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x134aea2]
|
mongod(+0xF49FF9) [0x1349ff9]
|
mongod(+0xF4A802) [0x134a802]
|
libpthread.so.0(+0xF100) [0x7fd053a26100]
|
libc.so.6(gsignal+0x37) [0x7fd05368b5f7]
|
libc.so.6(abort+0x148) [0x7fd05368cce8]
|
mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x12cb0fb]
|
mongod(_ZNK5mongo8Database13getCollectionENS_10StringDataE+0x42) [0xb0b062]
|
mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0x28) [0xb0b1d8]
|
mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x284) [0xb0cd74]
|
mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0x40C) [0xb10fec]
|
mongod(+0x5BE50A) [0x9be50a]
|
mongod(+0x5C114F) [0x9c114f]
|
mongod(main+0x15D) [0x97709d]
|
libc.so.6(__libc_start_main+0xF5) [0x7fd053677b15]
|
mongod(+0x5BC597) [0x9bc597]
|
----- END BACKTRACE -----
|
|
|
Just upgrade to version 3.0.7 and now the log looks as follows:
2015-11-19T09:26:33.346+0100 I CONTROL ***** SERVER RESTARTED *****
2015-11-19T09:26:33.405+0100 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2015-11-19T09:26:33.425+0100 I JOURNAL [initandlisten] journal dir=/var/lib/mongodb/journal
2015-11-19T09:26:33.425+0100 I JOURNAL [initandlisten] recover begin
2015-11-19T09:26:33.425+0100 I JOURNAL [initandlisten] info no lsn file in journal/ directory
2015-11-19T09:26:33.425+0100 I JOURNAL [initandlisten] recover lsn: 0
2015-11-19T09:26:33.425+0100 I JOURNAL [initandlisten] recover /var/lib/mongodb/journal/j._0
2015-11-19T09:26:33.426+0100 I JOURNAL [initandlisten] recover cleaning up
2015-11-19T09:26:33.426+0100 I JOURNAL [initandlisten] removeJournalFiles
2015-11-19T09:26:33.621+0100 I JOURNAL [initandlisten] recover done
2015-11-19T09:26:33.621+0100 I JOURNAL [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0
2015-11-19T09:26:39.143+0100 I JOURNAL [durability] Durability thread started
2015-11-19T09:26:39.144+0100 I JOURNAL [journal writer] Journal writer thread started
2015-11-19T09:26:39.146+0100 I CONTROL [initandlisten] MongoDB starting : pid=28783 port=27017 dbpath=/var/lib/mongodb 64-bit host=tobis
2015-11-19T09:26:39.147+0100 I CONTROL [initandlisten] db version v3.0.7
2015-11-19T09:26:39.147+0100 I CONTROL [initandlisten] git version: 6ce7cbe8c6b899552dadd907604559806aa2e9bd
2015-11-19T09:26:39.147+0100 I CONTROL [initandlisten] build info: Linux ip-10-229-88-125 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-11-19T09:26:39.147+0100 I CONTROL [initandlisten] allocator: tcmalloc
2015-11-19T09:26:39.147+0100 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:
{ bindIp: "127.0.0.1", port: 27017 }
, storage: { dbPath: "/var/lib/mongodb", journal:
{ enabled: true }
}, systemLog:
{ destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" }
}
2015-11-19T09:26:39.205+0100 I - [initandlisten] Invariant failure _name == nsToDatabaseSubstring(ns) src/mongo/db/catalog/database.cpp 419
2015-11-19T09:26:39.216+0100 I CONTROL [initandlisten]
0xf5c3e9 0xefb231 0xedfd52 0x91c662 0x91c75f 0x91e6d3 0x921240 0x8092a5 0x7d6db9 0x7f0d427b7ec5 0x806677
----- BEGIN BACKTRACE -----
{"backtrace":[
{"b":"400000","o":"B5C3E9"}
,
{"b":"400000","o":"AFB231"}
,
{"b":"400000","o":"ADFD52"}
,
{"b":"400000","o":"51C662"}
,
{"b":"400000","o":"51C75F"}
,
{"b":"400000","o":"51E6D3"}
,
{"b":"400000","o":"521240"}
,
{"b":"400000","o":"4092A5"}
,
{"b":"400000","o":"3D6DB9"}
,
{"b":"7F0D42796000","o":"21EC5"}
,
{"b":"400000","o":"406677"}
],"processInfo":{ "mongodbVersion" : "3.0.7", "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd", "uname" :
{ "sysname" : "Linux", "release" : "3.13.0-66-generic", "version" : "#108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015", "machine" : "x86_64" }
, "somap" : [
{ "elfType" : 2, "b" : "400000", "buildId" : "03104A2519911F189AABD1EBC9931E8EBE9AD44C" }
,
{ "b" : "7FFE798D1000", "elfType" : 3, "buildId" : "64F93A900F0AA19FFBFF98876568D341D09FF276" }
,
{ "b" : "7F0D43DC1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }
,
{ "b" : "7F0D43B62000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "A20EFFEC993A8441FA17F2079F923CBD04079E19" }
,
{ "b" : "7F0D43787000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F000D29917E9B6E94A35A8F02E5C62846E5916BC" }
,
{ "b" : "7F0D4357F000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }
,
{ "b" : "7F0D4337B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }
,
{ "b" : "7F0D43077000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4BF6F7ADD8244AD86008E6BF40D90F8873892197" }
,
{ "b" : "7F0D42D71000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }
,
{ "b" : "7F0D42B5B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }
,
{ "b" : "7F0D42796000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }
,
{ "b" : "7F0D43FDF000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" }
] }}
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5c3e9]
mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefb231]
mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xedfd52]
mongod(_ZNK5mongo8Database13getCollectionERKNS_10StringDataE+0x42) [0x91c662]
mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextERKNS_10StringDataE+0x1F) [0x91c75f]
mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextERKNS_10StringDataEPNS_20DatabaseCatalogEntryE+0x1E3) [0x91e6d3]
mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x150) [0x921240]
mongod(_ZN5mongo13initAndListenEi+0xE65) [0x8092a5]
mongod(main+0x139) [0x7d6db9]
libc.so.6(__libc_start_main+0xF5) [0x7f0d427b7ec5]
mongod(+0x406677) [0x806677]
----- END BACKTRACE -----
2015-11-19T09:26:39.216+0100 I - [initandlisten]
***aborting after invariant() failure
|
|
Hi!
I'm having exactly the same problem, however on an Ubuntu server, so it would be great if you could re-open this issue.
{{
2015-11-19T08:23:22.645+0100 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2015-11-19T08:23:22.676+0100 I JOURNAL [initandlisten] journal dir=/var/lib/mongodb/journal
2015-11-19T08:23:22.677+0100 I JOURNAL [initandlisten] recover begin
2015-11-19T08:23:22.677+0100 I JOURNAL [initandlisten] info no lsn file in journal/ directory
2015-11-19T08:23:22.677+0100 I JOURNAL [initandlisten] recover lsn: 0
2015-11-19T08:23:22.677+0100 I JOURNAL [initandlisten] recover /var/lib/mongodb/journal/j._0
2015-11-19T08:23:22.686+0100 I JOURNAL [initandlisten] recover cleaning up
2015-11-19T08:23:22.686+0100 I JOURNAL [initandlisten] removeJournalFiles
2015-11-19T08:23:22.959+0100 I JOURNAL [initandlisten] recover done
2015-11-19T08:23:22.959+0100 I JOURNAL [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0
2015-11-19T08:23:27.562+0100 I JOURNAL [durability] Durability thread started
2015-11-19T08:23:27.563+0100 I JOURNAL [journal writer] Journal writer thread started
2015-11-19T08:23:27.566+0100 I CONTROL [initandlisten] MongoDB starting : pid=1231 port=27017 dbpath=/var/lib/mongodb 64-bit host=tobis
2015-11-19T08:23:27.566+0100 I CONTROL [initandlisten] db version v3.0.5
2015-11-19T08:23:27.566+0100 I CONTROL [initandlisten] git version: 8bc4ae20708dbb493cb09338d9e7be6698e4a3a3
2015-11-19T08:23:27.566+0100 I CONTROL [initandlisten] build info: Linux ip-10-183-35-50 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-11-19T08:23:27.566+0100 I CONTROL [initandlisten] allocator: tcmalloc
2015-11-19T08:23:27.566+0100 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:
{ bindIp: "127.0.0.1" }
, storage:
{ dbPath: "/var/lib/mongodb" }
, systemLog:
{ destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" }
}
2015-11-19T08:23:27.582+0100 I - [initandlisten] Invariant failure details src/mongo/db/storage/mmap_v1/mmap_v1_database_catalog_entry.cpp 732
2015-11-19T08:23:27.595+0100 I CONTROL [initandlisten]
0xf5ba59 0xefa911 0xedf432 0xd3e0af 0xd3e65f 0xd3edcf 0xd41fff 0x921155 0x808f61 0x7d6ad9 0x7feb11e0cec5 0x806597
----- BEGIN BACKTRACE -----
{"backtrace":[
{"b":"400000","o":"B5BA59"}
,
{"b":"400000","o":"AFA911"}
,
{"b":"400000","o":"ADF432"}
,
{"b":"400000","o":"93E0AF"}
,
{"b":"400000","o":"93E65F"}
,
{"b":"400000","o":"93EDCF"}
,
{"b":"400000","o":"941FFF"}
,
{"b":"400000","o":"521155"}
,
{"b":"400000","o":"408F61"}
,
{"b":"400000","o":"3D6AD9"}
,
{"b":"7FEB11DEB000","o":"21EC5"}
,
{"b":"400000","o":"406597"}
],"processInfo":{ "mongodbVersion" : "3.0.5", "gitVersion" : "8bc4ae20708dbb493cb09338d9e7be6698e4a3a3", "uname" :
{ "sysname" : "Linux", "release" : "3.13.0-66-generic", "version" : "#108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015", "machine" : "x86_64" }
, "somap" : [
{ "elfType" : 2, "b" : "400000", "buildId" : "695FC6828398A9DB1F99718671147885B5ED116D" }
,
{ "b" : "7FFDBEA95000", "elfType" : 3, "buildId" : "64F93A900F0AA19FFBFF98876568D341D09FF276" }
,
{ "b" : "7FEB13416000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }
,
{ "b" : "7FEB131B7000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "A20EFFEC993A8441FA17F2079F923CBD04079E19" }
,
{ "b" : "7FEB12DDC000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F000D29917E9B6E94A35A8F02E5C62846E5916BC" }
,
{ "b" : "7FEB12BD4000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }
,
{ "b" : "7FEB129D0000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }
,
{ "b" : "7FEB126CC000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4BF6F7ADD8244AD86008E6BF40D90F8873892197" }
,
{ "b" : "7FEB123C6000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }
,
{ "b" : "7FEB121B0000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }
,
{ "b" : "7FEB11DEB000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }
,
{ "b" : "7FEB13634000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" }
] }}
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5ba59]
mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefa911]
mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xedf432]
mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry14_insertInCacheEPNS_16OperationContextERKNS_10StringDataEPNS0_5EntryE+0x32F) [0xd3e0af]
mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry5_initEPNS_16OperationContextE+0x4EF) [0xd3e65f]
mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntryC1EPNS_16OperationContextERKNS_10StringDataES5_bb+0x16F) [0xd3edcf]
mongod(_ZN5mongo12MMAPV1Engine23getDatabaseCatalogEntryEPNS_16OperationContextERKNS_10StringDataE+0x1AF) [0xd41fff]
mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x105) [0x921155]
mongod(_ZN5mongo13initAndListenEi+0xC01) [0x808f61]
mongod(main+0x139) [0x7d6ad9]
libc.so.6(__libc_start_main+0xF5) [0x7feb11e0cec5]
mongod(+0x406597) [0x806597]
----- END BACKTRACE -----
2015-11-19T08:23:27.595+0100 I - [initandlisten]
***aborting after invariant() failure
}}
/etc/mongod.conf
{{
- mongod.conf
- Where to store the data.
- Note: if you run mongodb as a non-root user (recommended) you may
- need to create and set permissions for this directory manually,
- e.g., if the parent directory isn't mutable by the mongodb user.
dbpath=/var/lib/mongodb
#where to log
logpath=/var/log/mongodb/mongod.log
logappend=true
#port = 27017
- Listen to local interface only. Comment out to listen on all interfaces.
bind_ip = 127.0.0.1
- Disables write-ahead journaling
- nojournal = true
- Enables periodic logging of CPU utilization and I/O wait
#cpu = true
- Turn on/off security. Off is currently the default
#noauth = true
#auth = true
- Verbose logging output.
#verbose = true
- Inspect all client data for validity on receipt (useful for
- developing drivers)
#objcheck = true
- Enable db quota management
#quota = true
- Set oplogging level where n is
- 0=off (default)
- 1=W
- 2=R
- 3=both
- 7=W+some reads
#diaglog = 0
- Ignore query hints
#nohints = true
- Enable the HTTP interface (Defaults to port 28017).
#httpinterface = true
- Turns off server-side scripting. This will result in greatly limited
- functionality
#noscripting = true
- Turns off table scans. Any query that would do a table scan fails.
#notablescan = true
- Disable data file preallocation.
#noprealloc = true
- Specify .ns file size for new databases.
- nssize = <size>
- Replication Options
- in replicated mongo databases, specify the replica set name here
#replSet=setname
- maximum size in megabytes for replication operation log
#oplogSize=1024
- path to a key file storing authentication info for connections
- between replica set members
#keyFile=/path/to/keyfile
}}
/etc/init/mongod.conf
{{
- Ubuntu upstart file at /etc/init/mongod.conf
- Recommended ulimit values for mongod or mongos
- See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
#
limit fsize unlimited unlimited
limit cpu unlimited unlimited
limit as unlimited unlimited
limit nofile 64000 64000
limit rss unlimited unlimited
limit nproc 32000 32000
kill timeout 300 # wait 300s between SIGTERM and SIGKILL.
pre-start script
DAEMONUSER=${DAEMONUSER:-mongodb}
if [ ! -d /var/lib/mongod ]; then
mkdir -p /var/lib/mongodb && chown mongodb:mongodb /var/lib/mongodb
fi
if [ ! -d /var/log/mongod ]; then
mkdir -p /var/log/mongodb && chown mongodb:mongodb /var/log/mongodb
fi
touch /var/run/mongodb.pid
chown $DAEMONUSER /var/run/mongodb.pid
end script
start on runlevel [2345]
stop on runlevel [06]
script
ENABLE_MONGOD="yes"
CONF=/etc/mongod.conf
DAEMON=/usr/bin/mongod
DAEMONUSER=${DAEMONUSER:-mongodb}
if [ -f /etc/default/mongod ]; then . /etc/default/mongod; fi
- Handle NUMA access to CPUs (
SERVER-3574)
- This verifies the existence of numactl as well as testing that the command works
NUMACTL_ARGS="--interleave=all"
if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null
then
NUMACTL="$(which numactl) – $NUMACTL_ARGS"
DAEMON_OPTS=${DAEMON_OPTS:"-config $CONF"}
else
NUMACTL=""
DAEMON_OPTS="-- "${DAEMON_OPTS:"-config $CONF"}
fi
if [ "x$ENABLE_MONGOD" = "xyes" ]
then
exec start-stop-daemon --start \
--chuid $DAEMONUSER \
--pidfile /var/run/mongodb.pid \
--make-pidfile \
--exec $NUMACTL $DAEMON $DAEMON_OPTS
fi
end script
}}
Listing of dbpath:
{{
drwxr-xr-x 3 mongodb mongodb 4096 Jul 29 08:12 .
drwxr-xr-x 41 root root 4096 Jul 29 06:35 ..
rw------ 1 mongodb mongodb 67108864 Nov 4 13:25 DigiCoaching.0
rw------ 1 mongodb mongodb 134217728 Jul 28 15:26 DigiCoaching.1
rw------ 1 mongodb mongodb 16777216 Nov 4 12:22 DigiCoaching.ns
drwxr-xr-x 2 mongodb mongodb 4096 Nov 19 08:23 journal
rw------ 1 mongodb mongodb 67108864 Jul 29 08:12 local.0
rw------ 1 mongodb mongodb 16777216 Jul 29 08:12 local.ns
-rwxr-xr-x 1 mongodb mongodb 5 Nov 19 08:23 mongod.lock
rw-rr- 1 mongodb nogroup 69 Jul 29 08:12 storage.bson
}}
Starting mongod with -vvvvv:
{{
/var/lib$ sudo mongod -vvvvvv -dbpath /var/lib/mongodb
2015-11-19T08:45:11.082+0100 D SHARDING isInRangeTest passed
2015-11-19T08:45:11.083+0100 D NETWORK [initandlisten] fd limit hard:4096 soft:1024 max conn: 819
2015-11-19T08:45:11.136+0100 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2015-11-19T08:45:11.152+0100 I JOURNAL [initandlisten] journal dir=/var/lib/mongodb/journal
2015-11-19T08:45:11.153+0100 D COMMAND [DataFileSync] BackgroundJob starting: DataFileSync
2015-11-19T08:45:11.153+0100 I JOURNAL [initandlisten] recover begin
2015-11-19T08:45:11.153+0100 I JOURNAL [initandlisten] info no lsn file in journal/ directory
2015-11-19T08:45:11.153+0100 I JOURNAL [initandlisten] recover lsn: 0
2015-11-19T08:45:11.153+0100 I JOURNAL [initandlisten] recover /var/lib/mongodb/journal/j._0
2015-11-19T08:45:11.159+0100 I JOURNAL [initandlisten] recover cleaning up
2015-11-19T08:45:11.159+0100 I JOURNAL [initandlisten] removeJournalFiles
2015-11-19T08:45:11.385+0100 D STORAGE [initandlisten] flushing directory /var/lib/mongodb/journal
2015-11-19T08:45:11.400+0100 D JOURNAL [initandlisten] removeJournalFiles end
2015-11-19T08:45:11.400+0100 I JOURNAL [initandlisten] recover done
2015-11-19T08:45:11.400+0100 I JOURNAL [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0
2015-11-19T08:45:15.855+0100 D STORAGE [initandlisten] flushing directory /var/lib/mongodb/journal
2015-11-19T08:45:15.869+0100 D STORAGE [initandlisten] flushing directory /var/lib/mongodb/journal
2015-11-19T08:45:15.872+0100 I JOURNAL [durability] Durability thread started
2015-11-19T08:45:15.873+0100 I JOURNAL [journal writer] Journal writer thread started
2015-11-19T08:45:15.877+0100 D STORAGE [initandlisten] flushing directory /var/lib/mongodb
2015-11-19T08:45:15.877+0100 I CONTROL [initandlisten] MongoDB starting : pid=1369 port=27017 dbpath=/var/lib/mongodb 64-bit host=tobis
2015-11-19T08:45:15.877+0100 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-11-19T08:45:15.877+0100 I CONTROL [initandlisten]
2015-11-19T08:45:15.877+0100 I CONTROL [initandlisten] db version v3.0.5
2015-11-19T08:45:15.877+0100 I CONTROL [initandlisten] git version: 8bc4ae20708dbb493cb09338d9e7be6698e4a3a3
2015-11-19T08:45:15.877+0100 I CONTROL [initandlisten] build info: Linux ip-10-183-35-50 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-11-19T08:45:15.877+0100 I CONTROL [initandlisten] allocator: tcmalloc
2015-11-19T08:45:15.877+0100 I CONTROL [initandlisten] options: { storage:
{ dbPath: "/var/lib/mongodb" }
, systemLog:
{ verbosity: 6 }
}
2015-11-19T08:45:15.877+0100 D STORAGE [initandlisten] enter repairDatabases (to check pdfile version #)
2015-11-19T08:45:15.877+0100 D STORAGE [initandlisten] Recovering database: local
2015-11-19T08:45:15.877+0100 D STORAGE [initandlisten] mmf open /var/lib/mongodb/local.ns
2015-11-19T08:45:15.878+0100 D STORAGE [initandlisten] mmf finishOpening 0x7fc4e20e0000 /var/lib/mongodb/local.ns len:16777216
2015-11-19T08:45:15.878+0100 D STORAGE [initandlisten] mmf open /var/lib/mongodb/local.0
2015-11-19T08:45:15.878+0100 D STORAGE [initandlisten] mmf finishOpening 0x7fc4dd0e0000 /var/lib/mongodb/local.0 len:67108864
2015-11-19T08:45:15.884+0100 D STORAGE [initandlisten] ***** ROLLING BACK 0 disk writes
2015-11-19T08:45:15.884+0100 D STORAGE [initandlisten] ***** ROLLING BACK 0 custom changes
2015-11-19T08:45:15.884+0100 D STORAGE [initandlisten] local.startup_log: clearing plan cache - collection info cache reset
2015-11-19T08:45:15.884+0100 D STORAGE [initandlisten] local.system.indexes: clearing plan cache - collection info cache reset
2015-11-19T08:45:15.884+0100 D STORAGE [initandlisten] local.system.namespaces: clearing plan cache - collection info cache reset
2015-11-19T08:45:15.885+0100 D STORAGE [initandlisten] Recovering database: DigiCoaching
2015-11-19T08:45:15.885+0100 D STORAGE [initandlisten] mmf open /var/lib/mongodb/DigiCoaching.ns
2015-11-19T08:45:15.885+0100 D STORAGE [initandlisten] mmf finishOpening 0x7fc4d80e0000 /var/lib/mongodb/DigiCoaching.ns len:16777216
2015-11-19T08:45:15.885+0100 D STORAGE [initandlisten] mmf open /var/lib/mongodb/DigiCoaching.0
2015-11-19T08:45:15.885+0100 D STORAGE [initandlisten] mmf finishOpening 0x7fc4d30e0000 /var/lib/mongodb/DigiCoaching.0 len:67108864
2015-11-19T08:45:15.891+0100 I - [initandlisten] Invariant failure details src/mongo/db/storage/mmap_v1/mmap_v1_database_catalog_entry.cpp 732
2015-11-19T08:45:15.904+0100 I CONTROL [initandlisten]
0xf5ba59 0xefa911 0xedf432 0xd3e0af 0xd3e65f 0xd3edcf 0xd41fff 0x921155 0x808f61 0x7d6ad9 0x7fc4e6544ec5 0x806597
----- BEGIN BACKTRACE -----
{"backtrace":[
{"b":"400000","o":"B5BA59"}
,
{"b":"400000","o":"AFA911"}
,
{"b":"400000","o":"ADF432"}
,
{"b":"400000","o":"93E0AF"}
,
{"b":"400000","o":"93E65F"}
,
{"b":"400000","o":"93EDCF"}
,
{"b":"400000","o":"941FFF"}
,
{"b":"400000","o":"521155"}
,
{"b":"400000","o":"408F61"}
,
{"b":"400000","o":"3D6AD9"}
,
{"b":"7FC4E6523000","o":"21EC5"}
,
{"b":"400000","o":"406597"}
],"processInfo":{ "mongodbVersion" : "3.0.5", "gitVersion" : "8bc4ae20708dbb493cb09338d9e7be6698e4a3a3", "uname" :
{ "sysname" : "Linux", "release" : "3.13.0-66-generic", "version" : "#108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015", "machine" : "x86_64" }
, "somap" : [
{ "elfType" : 2, "b" : "400000", "buildId" : "695FC6828398A9DB1F99718671147885B5ED116D" }
,
{ "b" : "7FFD975A5000", "elfType" : 3, "buildId" : "64F93A900F0AA19FFBFF98876568D341D09FF276" }
,
{ "b" : "7FC4E7B4E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }
,
{ "b" : "7FC4E78EF000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "A20EFFEC993A8441FA17F2079F923CBD04079E19" }
,
{ "b" : "7FC4E7514000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F000D29917E9B6E94A35A8F02E5C62846E5916BC" }
,
{ "b" : "7FC4E730C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }
,
{ "b" : "7FC4E7108000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }
,
{ "b" : "7FC4E6E04000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4BF6F7ADD8244AD86008E6BF40D90F8873892197" }
,
{ "b" : "7FC4E6AFE000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }
,
{ "b" : "7FC4E68E8000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }
,
{ "b" : "7FC4E6523000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }
,
{ "b" : "7FC4E7D6C000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" }
] }}
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5ba59]
mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefa911]
mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xedf432]
mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry14_insertInCacheEPNS_16OperationContextERKNS_10StringDataEPNS0_5EntryE+0x32F) [0xd3e0af]
mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry5_initEPNS_16OperationContextE+0x4EF) [0xd3e65f]
mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntryC1EPNS_16OperationContextERKNS_10StringDataES5_bb+0x16F) [0xd3edcf]
mongod(_ZN5mongo12MMAPV1Engine23getDatabaseCatalogEntryEPNS_16OperationContextERKNS_10StringDataE+0x1AF) [0xd41fff]
mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x105) [0x921155]
mongod(_ZN5mongo13initAndListenEi+0xC01) [0x808f61]
mongod(main+0x139) [0x7d6ad9]
libc.so.6(__libc_start_main+0xF5) [0x7fc4e6544ec5]
mongod(+0x406597) [0x806597]
----- END BACKTRACE -----
2015-11-19T08:45:15.904+0100 I - [initandlisten]
***aborting after invariant() failure
}}
Server is standalone
|
|
Ok, I understand, thank you for the answers. If it happens again, I'll be sure to let you know (and will definitely do my best to keep dbpath intact).
|
|
reunanen, if this happens again the directory listing may not be sufficient. To find out where the issue happened we may need to look at the structure of the file to determine if it was a filesystem problem or a MongoDB problem.
We understand if data can't be shared publicly. One alternative is a private upload portal, where data can be made available only to MongoDB engineers working on the problem. If this is still not an option, we could send instructions on how to find out the exact nature of the file corruption. In any event, you may need to keep the contents of dbpath around.
With respect to WiredTiger, early in the 3.0 release cycle we identified and fix various corruption issues caused by unclean shutdowns, but 3.0.5 includes fixes for all of them. It's hard for me to say if WiredTiger would behave better in this particular case if the problem was at the filesystem level – unless the filesystem provides certain guarantees there's not much any storage engine can do to guard against some of these issues, but WiredTiger in general has better recovery mechanisms.
Note that one can run mixed storage engine replica sets, so if unclean shutdowns are common in this environment you could try adding a hidden node running WiredTiger to your replica set for evaluation purposes.
Regards,
Ramón.
|
|
Thanks for the comment, Ramon. Yes I understand perfectly, and I agree that this is the best way to move on.
I have one more question, though: in case it happens again, do I understand correctly that you need just the directory listing, i.e. not the files themselves? (Well I probably can't send the file contents anyway; the data is not really mine, but belongs to a customer...)
Also, would you assume that WiredTiger is more resilient than MMAPv1 against these kinds of conditions – in particular on Windows?
|
|
Thanks for the additional information reunanen. Unfortunately, without being able to examine the contents of dbpath it's not possible to determine how some file contents where corrupted on that unclean shutdown. While the journal should make sure your data can be recovered, the OS may not have been able to keep the consistency of some of the open files and, upon reboot, decided to "fix" them in a way that left them inconsistent for mongod to use.
I understand that the top priority for you was to get this system back up and running, but thanks for taking the time to open ticket and log the details.
We're closing this ticket for now due to the lack of debugging information, but if you run into this again feel free to post a note here so we can reopen (or open a new one); saving the contents of dbpath should allow us to investigate further should you run into this issue again.
Regards,
Ramón.
|
|
Hi Dan,
Thank you for your reply.
I already had to recover the live production system by deleting the data directory, and restoring the critical data from a backup. So now the system is up and running again, but I cannot provide all the answers for debugging purposes any longer...
- I did not see the shutdown myself, but basically it was a normal power outage on the server. In the Windows Event Viewer log, I have: The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly. (I know it was a power outage and not blue screen because many other PCs had the same at the same time. It's an industrial site, and such outages are not extremely uncommon... Yes there should be UPS, but frankly the application is such that if the last 5 s or even 5 min of data from before the outage are lost, that's no problem at all, as long as the system recovers and reads the critical data (which does not change often). As far as I can understand, journaling is enabled as default behavior, so this should not be a problem.)
- Attached.
- Sorry – as I wrote above, the directory contents are gone now. Should have taken a copy of the data directory first of course, but my number one priority really was getting the whole application up and running again (and I knew I had a good backup of the database). But I can try to do this, if the problem occurs again.
- Same here: can't do it anymore, but will try if it happens again. Didn't know about this option, unfortunately. Actually not too familiar with MongoDB in general, but learning more all the time. 8)
- It's a standalone server. And there's not really anything special – basically running a CRUD application with default settings. The volume of incoming (non-critical) data flow is consistently (relatively) high, though.
Since I cannot tell you how to reproduce the problem, I don't mind if you just close this ticket. Frankly, I decided to add the ticket mainly because a Google search on "invariant failure details" gave zero exact matches. This was quite surprising to me, so I figured that if someone else will have the same problem later, at least they should be able to find this discussion... and in the best case, they can perhaps provide the missing pieces of information.
For what it's worth, I also found this in the Windows Event Viewer log right after the PC booted (but don't know if it's related or not):
The MongoDB service terminated with service-specific error Cannot create another system semaphore..
|
|
Hi Juha,
From the log file, the journal applied cleanly and the system was doing some pre-startup checks of the catalog when it hit the invariant.
A couple questions:
- Can you say how the original unclean shutdown occurred?
- Can you attach the server config file that is located here: C:\Program Files\MongoDB\Server\3.0\bin\mongod.cfg
- Can you list the directory contents of the dbpath?
- Can you try restarting with extra verbosity and attach the result? Add -vvvvv to the command line or set in the config file. Here's the docs for instructions.
- Is this server part of a replica set or a standalone server?
|
Generated at Thu Feb 08 03:53:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.