[SERVER-52753] Too many open files error on mongo shell start attempt Created: 10/Nov/20 Updated: 17/Dec/20 Resolved: 17/Dec/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Alex TSM TSM | Assignee: | Edwin Zhou |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Participants: |
| Description |
|
I am on a mac starting mongo from terminal via command: sudo mongod -storageEngine wiredTiger --directoryperdb --dbpath /mongodb/data --setParameter failIndexKeyTooLong=false
And I get:
2020-11-11T01:12:59.114+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2020-11-11T01:13:04.809+0200 E STORAGE [initandlisten] WiredTiger (24) [1605049984:809579][2786:0x7fff768bd300], file:e3/collection-917--6708257915232949607.wt, session.open_cursor: /Users/x/mongodb/data/e3/collection-917--6708257915232949607.wt: Too many open files 2020-11-11T01:13:04.809+0200 I - [initandlisten] Invariant failure: ret resulted in status UnknownError 24: Too many open files at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 78 2020-11-11T01:13:04.816+0200 I CONTROL [initandlisten] 0x10bf8ad79 0x10bf3e5a0 0x10bf2ba25 0x10bdba9a9 0x10bdb94e4 0x10bdb5511 0x10bdb32cc 0x10bdaff6a 0x10bdac7e4 0x10bd28168 0x10bd2afa9 0x10bdaa11c 0x10bab88cc 0x10b81a7c8 0x10b819f53 0x10b81ee9c 0x10b819f34 ----- BEGIN BACKTRACE ----- {"backtrace":[\{"b":"10B819000","o":"771D79"},{"b":"10B819000","o":"7255A0"},{"b":"10B819000","o":"712A25"},{"b":"10B819000","o":"5A19A9"},{"b":"10B819000","o":"5A04E4"},{"b":"10B819000","o":"59C511"},{"b":"10B819000","o":"59A2CC"},{"b":"10B819000","o":"596F6A"},{"b":"10B819000","o":"5937E4"},{"b":"10B819000","o":"50F168"},{"b":"10B819000","o":"511FA9"},{"b":"10B819000","o":"59111C"},{"b":"10B819000","o":"29F8CC"},{"b":"10B819000","o":"17C8"},{"b":"10B819000","o":"F53"},{"b":"10B819000","o":"5E9C"},{"b":"10B819000","o":"F34"}],"processInfo":{ "mongodbVersion" : "3.0.1", "gitVersion" : "534b5a3f9d10f00cd27737fbcd951032248b5952", "uname" : { "sysname" : "Darwin", "release" : "14.5.0", "version" : "Darwin Kernel Version 14.5.0: Sun Jun 4 21:40:08 PDT 2017; root:xnu-2782.70.3~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/alex_fimm_dev/Desktop/Databases/Mongo/mongodb/bin/mongod", "machType" : 2, "b" : "10B819000", "buildId" : "BCCE8C3AD8BB31D490CB01C153674428" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF93915000", "buildId" : "4B24B5BE45F2355A9A35D438A516D140" }, { "path" : "/usr/lib/libc+.1.dylib", "machType" : 6, "b" : "7FFF9029B000", "buildId" : "1B9530FD989B3174BB1CBDC159501710" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF93298000", "buildId" : "45E9A2E799C436B2BEE30C4E11614AD1" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF863A8000", "buildId" : "E789748DF9A73CFFB31790DF348B1E95" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF8665E000", "buildId" : "BF8FC133EE103DA69B9092039E28678F" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF9449B000", "buildId" : "0C68D3A6ACDD3EF3991ACC82C32AB836" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF93A8D000", "buildId" : "5779FFA04D9A3AD4B7F2618227621DC8" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF8B059000", "buildId" : "A61E703C784A3698B51375DD12AAD6DC" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF9013A000", "buildId" : "224F35C56E353EC4A97C80949CE444E7" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8EC4B000", "buildId" : "77845842DE703CC5BD01C3D14227CED5" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF941CA000", "buildId" : "4F81CA3AD2CE3030A89D42F3DAD7BA8F" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF8DF4C000", "buildId" : "126CA2EDDE91308F8881B9DAEC3C63B6" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF867C8000", "buildId" : "7AF900412768378A925AD83161863642" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF8B0F7000", "buildId" : "3485B5F46CE83C628DFD8736ED6E8531" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF8BAC5000", "buildId" : "F153AC5B0542356E88C820A62CA704E2" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF8E4CC000", "buildId" : "9615D10AFCA73BE4AA1A1B195DACE1A1" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF8BCB7000", "buildId" : "2E673D96E33437D2833AE57383BBA377" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF8E667000", "buildId" : "56F94DCEDBDE36158F07DE6270D9F8BE" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF8722A000", "buildId" : "41B7C5785A5331C8A96FC73E030B0938" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF8664C000", "buildId" : "155DA0A92046332EBFA3D7974A51F731" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF85BA3000", "buildId" : "9EC5AF92D0D23BDE92B6D3730D3865C8" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF86B11000", "buildId" : "2E16C4B3A32739579C41143911979A1E" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF9238B000", "buildId" : "79B806D6AEEF3663BAE32D8A4A0A7A92" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF921CA000", "buildId" : "1E12AB456D9636D0A226F24D9FB0D9D6" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF89697000", "buildId" : "DDA8928BCC0D3255BD8A3FEA0982B890" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8E046000", "buildId" : "6105C13467223C0AA4CE5E1261E2E1CC" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF9225B000", "buildId" : "BA58B30B83773B0A8AE34F84021D9D4E" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF8B3E6000", "buildId" : "61147800F3203DAA850CBADF33855F29" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF90181000", "buildId" : "64E34079D7123D669CE2418624A5C040" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8BA04000", "buildId" : "ACE90967ECD03251AEEB461E3C6414F7" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF921B3000", "buildId" : "3F5E973FC70231AC97BC05F5C195683C" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF8B084000", "buildId" : "581DAD0F6B633A48B63B917AF799ABAA" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF93004000", "buildId" : "4F79A38D3C9C35788E387697F7300CB2" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF91A51000", "buildId" : "840F5301B55A307890B9FEFFD6CD741A" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF86BE7000", "buildId" : "5676F7EAC1DF329FB006D2C3022B7D70" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF90598000", "buildId" : "BE7E51A0B6EA3A549CCA9D88F683A6D6" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF9260D000", "buildId" : "5C829202962E37448B5000D38CC88E84" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF8F704000", "buildId" : "759E155DBC423D4E869B6F57D477177C" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF92FBD000", "buildId" : "A260789BD4D8316A9490254767B8A5F1" }, { "path" : "/usr/lib/libc+abi.dylib", "machType" : 6, "b" : "7FFF89653000", "buildId" : "B8E6BB31E27E3B809BA273B5AB7E42E1" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF8E07F000", "buildId" : "2EE8E4365CDC34C599595BA218D507FB" } ] }} mongod(ZN5mongo15printStackTraceERNSt3_113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x10bf8ad79] mongod(_ZN5mongo10logContextEPKc+0x100) [0x10bf3e5a0] mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x175) [0x10bf2ba25] mongod(ZN5mongo17WiredTigerSession9getCursorERKNSt3_112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEyb+0x1A9) [0x10bdba9a9] mongod(ZN5mongo16WiredTigerCursorC2ERKNSt3_112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEybPNS_16OperationContextE+0x84) [0x10bdb94e4] mongod(_ZN5mongo21WiredTigerRecordStore8IteratorC2ERKS0_PNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionEb+0x61) [0x10bdb5511] mongod(_ZNK5mongo21WiredTigerRecordStore11getIteratorEPNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionE+0x6C) [0x10bdb32cc] mongod(_ZN5mongo21WiredTigerRecordStoreC2EPNS_16OperationContextERKNS_10StringDataES5_bxxPNS_28CappedDocumentDeleteCallbackEPNS_20WiredTigerSizeStorerE+0x38A) [0x10bdaff6a] mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextERKNS_10StringDataES5_RKNS_17CollectionOptionsE+0x134) [0x10bdac7e4] mongod(ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKNSt3_112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEb+0x158) [0x10bd28168] mongod(_ZN5mongo15KVStorageEngineC2EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x919) [0x10bd2afa9] mongod(ZNK5mongo12_GLOBAL_N_117WiredTigerFactory6createERKNS_19StorageGlobalParamsERKNS_21StorageEngineLockFileE+0x11C) [0x10bdaa11c] mongod(ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKNSt3_112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE+0x27C) [0x10bab88cc] mongod(_ZN5mongoL14_initAndListenEi+0x328) [0x10b81a7c8] mongod(_ZN5mongo13initAndListenEi+0x13) [0x10b819f53] mongod(main+0x3BC) [0x10b81ee9c] mongod(start+0x34) [0x10b819f34] ----- END BACKTRACE ----- 2020-11-11T01:13:04.817+0200 I - [initandlisten]
***aborting after invariant() failure
Any help would be be appreciated, thanks! |
| Comments |
| Comment by Edwin Zhou [ 17/Dec/20 ] |
|
Thank you for providing the files. Unfortunately I wasn't able to repair this corruption, and it can be an issue with 3.0. Given that you're operating on an unsupported version of MongoDB, I advise that you upgrade to MongoDB 3.6 or later. To avoid a problem like this in the future, it is our strong recommendation to:
Best, |
| Comment by Alex TSM TSM [ 17/Dec/20 ] |
|
Hello Edwin, My sincere apology for not responding. I have uploaded the files and will respond faster in the future. Thank you for all the effort! Br, |
| Comment by Edwin Zhou [ 17/Dec/20 ] |
|
Would you like to continue with the metadata repair? If this is still an issue for you, would you please upload wiredTiger.wt and wiredTiger.turtle to the secure upload portal? Thanks, |
| Comment by Edwin Zhou [ 02/Dec/20 ] |
|
I appreciate your concern. The files I requested only contain metadata about other *.wt files but not the data stored in the database. Collection and index data is stored in collection-*.wt and index-*.wt files respectively, which I have not requested. Best, |
| Comment by Alex TSM TSM [ 02/Dec/20 ] |
|
Hello, First of all thank you for the help, very much appreciated. Is any of the information stored in the database exposed through the requested files?
Best regards, Alex |
| Comment by Edwin Zhou [ 02/Dec/20 ] |
|
As MongoDB 3.0.1 is EOL, we can offer a one time effort to help and try to repair the .wt files. Please attach copies of the wiredTiger.wt and wiredTiger.turtle files to the upload portal and we can attempt a metadata-only repair effort using internal tools. Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Keep in mind that this repair effort may not be successful, and that diagnosing corruption issues requires significant information and effort. Kind regards, |
| Comment by Alex TSM TSM [ 28/Nov/20 ] |
|
Logs for start: _____________________ sudo mongod -storageEngine wiredTiger --directoryperdb --dbpath /Users/alex_fimm_dev/mongodb/data --setParameter failIndexKeyTooLong=false _____________________
2020-11-28T14:21:39.615+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2020-11-28T14:21:40.124+0200 I CONTROL [initandlisten] MongoDB starting : pid=53363 port=27017 dbpath=/Users/alex/mongodb/data 64-bit host=lm50G3QD 2020-11-28T14:21:40.124+0200 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. 2020-11-28T14:21:40.124+0200 I CONTROL [initandlisten] 2020-11-28T14:21:40.124+0200 I CONTROL [initandlisten] db version v3.0.1 2020-11-28T14:21:40.124+0200 I CONTROL [initandlisten] git version: 534b5a3f9d10f00cd27737fbcd951032248b5952 2020-11-28T14:21:40.124+0200 I CONTROL [initandlisten] build info: Darwin bs-osx108-7 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 2020-11-28T14:21:40.124+0200 I CONTROL [initandlisten] allocator: system 2020-11-28T14:21:40.124+0200 I CONTROL [initandlisten] options: { setParameter: { failIndexKeyTooLong: "false" }, storage: { dbPath: "/Users/alex/mongodb/data", directoryPerDB: true, engine: "wiredTiger" } } 2020-11-28T14:21:40.138+0200 I NETWORK [initandlisten] waiting for connections on port 27017
_____________________
How can I get the logs from repair?
Is there no way of restoring the database form the folder containing the .wt files ?
Thanks again for the help and sorry for late reply
|
| Comment by Edwin Zhou [ 18/Nov/20 ] |
|
Thanks for informing us of the steps you took. Could you attach the following:
Best, Edwin |
| Comment by Alex TSM TSM [ 18/Nov/20 ] |
|
Hello and thanks for your response. I ran mongod with --repair flag successfully and then fired mongod normally successfully as well. Problem now is that when I launch 'mongo' in another terminal and run the 'show dbs' not all show up and they all show 0 mb in size. When I switch to the database that I am interested in (which does appear but size 0 as well) and run the 'show collections' command they are all missing. When I check the data directory the file with the folder with the database name is there and all .wt files are there and taking a total space of 7gb. Is there any diagnosis command I can run to repair or create a new database using the existing broken database . I do not have a dump... Any advice? |
| Comment by Edwin Zhou [ 13/Nov/20 ] |
|
Can you validate your ulimit settings using ulimit -a and checking them against the recommended ulimit settings for 3.0? Best, Edwin |
| Comment by Alex TSM TSM [ 10/Nov/20 ] |
|
Using MongoDB shell version: 3.0.1 |