[SERVER-52753] Too many open files error on mongo shell start attempt Created: 10/Nov/20  Updated: 17/Dec/20  Resolved: 17/Dec/20

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alex TSM TSM Assignee: Edwin Zhou
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

I am on a mac starting mongo from terminal via command:

sudo mongod -storageEngine wiredTiger --directoryperdb --dbpath /mongodb/data --setParameter failIndexKeyTooLong=false

 

And I get:

 

2020-11-11T01:12:59.114+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),

2020-11-11T01:13:04.809+0200 E STORAGE  [initandlisten] WiredTiger (24) [1605049984:809579][2786:0x7fff768bd300], file:e3/collection-917--6708257915232949607.wt, session.open_cursor: /Users/x/mongodb/data/e3/collection-917--6708257915232949607.wt: Too many open files

2020-11-11T01:13:04.809+0200 I -        [initandlisten] Invariant failure: ret resulted in status UnknownError 24: Too many open files at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 78

2020-11-11T01:13:04.816+0200 I CONTROL  [initandlisten]

0x10bf8ad79 0x10bf3e5a0 0x10bf2ba25 0x10bdba9a9 0x10bdb94e4 0x10bdb5511 0x10bdb32cc 0x10bdaff6a 0x10bdac7e4 0x10bd28168 0x10bd2afa9 0x10bdaa11c 0x10bab88cc 0x10b81a7c8 0x10b819f53 0x10b81ee9c 0x10b819f34

----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"10B819000","o":"771D79"}

,{"b":"10B819000","o":"7255A0"},{"b":"10B819000","o":"712A25"},{"b":"10B819000","o":"5A19A9"},{"b":"10B819000","o":"5A04E4"},{"b":"10B819000","o":"59C511"},{"b":"10B819000","o":"59A2CC"},{"b":"10B819000","o":"596F6A"},{"b":"10B819000","o":"5937E4"},{"b":"10B819000","o":"50F168"},{"b":"10B819000","o":"511FA9"},{"b":"10B819000","o":"59111C"},{"b":"10B819000","o":"29F8CC"},{"b":"10B819000","o":"17C8"},{"b":"10B819000","o":"F53"},{"b":"10B819000","o":"5E9C"},{"b":"10B819000","o":"F34"}],"processInfo":{ "mongodbVersion" : "3.0.1", "gitVersion" : "534b5a3f9d10f00cd27737fbcd951032248b5952", "uname" :

{ "sysname" : "Darwin", "release" : "14.5.0", "version" : "Darwin Kernel Version 14.5.0: Sun Jun  4 21:40:08 PDT 2017; root:xnu-2782.70.3~1/RELEASE_X86_64", "machine" : "x86_64" }

, "somap" : [ { "path" : "/Users/alex_fimm_dev/Desktop/Databases/Mongo/mongodb/bin/mongod", "machType" : 2, "b" : "10B819000", "buildId" : "BCCE8C3AD8BB31D490CB01C153674428" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF93915000", "buildId" : "4B24B5BE45F2355A9A35D438A516D140" }, { "path" : "/usr/lib/libc+.1.dylib", "machType" : 6, "b" : "7FFF9029B000", "buildId" : "1B9530FD989B3174BB1CBDC159501710" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF93298000", "buildId" : "45E9A2E799C436B2BEE30C4E11614AD1" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF863A8000", "buildId" : "E789748DF9A73CFFB31790DF348B1E95" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF8665E000", "buildId" : "BF8FC133EE103DA69B9092039E28678F" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF9449B000", "buildId" : "0C68D3A6ACDD3EF3991ACC82C32AB836" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF93A8D000", "buildId" : "5779FFA04D9A3AD4B7F2618227621DC8" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF8B059000", "buildId" : "A61E703C784A3698B51375DD12AAD6DC" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF9013A000", "buildId" : "224F35C56E353EC4A97C80949CE444E7" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8EC4B000", "buildId" : "77845842DE703CC5BD01C3D14227CED5" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF941CA000", "buildId" : "4F81CA3AD2CE3030A89D42F3DAD7BA8F" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF8DF4C000", "buildId" : "126CA2EDDE91308F8881B9DAEC3C63B6" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF867C8000", "buildId" : "7AF900412768378A925AD83161863642" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF8B0F7000", "buildId" : "3485B5F46CE83C628DFD8736ED6E8531" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF8BAC5000", "buildId" : "F153AC5B0542356E88C820A62CA704E2" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF8E4CC000", "buildId" : "9615D10AFCA73BE4AA1A1B195DACE1A1" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF8BCB7000", "buildId" : "2E673D96E33437D2833AE57383BBA377" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF8E667000", "buildId" : "56F94DCEDBDE36158F07DE6270D9F8BE" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF8722A000", "buildId" : "41B7C5785A5331C8A96FC73E030B0938" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF8664C000", "buildId" : "155DA0A92046332EBFA3D7974A51F731" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF85BA3000", "buildId" : "9EC5AF92D0D23BDE92B6D3730D3865C8" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF86B11000", "buildId" : "2E16C4B3A32739579C41143911979A1E" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF9238B000", "buildId" : "79B806D6AEEF3663BAE32D8A4A0A7A92" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF921CA000", "buildId" : "1E12AB456D9636D0A226F24D9FB0D9D6" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF89697000", "buildId" : "DDA8928BCC0D3255BD8A3FEA0982B890" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8E046000", "buildId" : "6105C13467223C0AA4CE5E1261E2E1CC" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF9225B000", "buildId" : "BA58B30B83773B0A8AE34F84021D9D4E" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF8B3E6000", "buildId" : "61147800F3203DAA850CBADF33855F29" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF90181000", "buildId" : "64E34079D7123D669CE2418624A5C040" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8BA04000", "buildId" : "ACE90967ECD03251AEEB461E3C6414F7" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF921B3000", "buildId" : "3F5E973FC70231AC97BC05F5C195683C" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF8B084000", "buildId" : "581DAD0F6B633A48B63B917AF799ABAA" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF93004000", "buildId" : "4F79A38D3C9C35788E387697F7300CB2" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF91A51000", "buildId" : "840F5301B55A307890B9FEFFD6CD741A" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF86BE7000", "buildId" : "5676F7EAC1DF329FB006D2C3022B7D70" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF90598000", "buildId" : "BE7E51A0B6EA3A549CCA9D88F683A6D6" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF9260D000", "buildId" : "5C829202962E37448B5000D38CC88E84" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF8F704000", "buildId" : "759E155DBC423D4E869B6F57D477177C" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF92FBD000", "buildId" : "A260789BD4D8316A9490254767B8A5F1" }, { "path" : "/usr/lib/libc+abi.dylib", "machType" : 6, "b" : "7FFF89653000", "buildId" : "B8E6BB31E27E3B809BA273B5AB7E42E1" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF8E07F000", "buildId" : "2EE8E4365CDC34C599595BA218D507FB" } ] }}

mongod(ZN5mongo15printStackTraceERNSt3_113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x10bf8ad79]

mongod(_ZN5mongo10logContextEPKc+0x100) [0x10bf3e5a0]

mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x175) [0x10bf2ba25]

mongod(ZN5mongo17WiredTigerSession9getCursorERKNSt3_112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEyb+0x1A9) [0x10bdba9a9]

mongod(ZN5mongo16WiredTigerCursorC2ERKNSt3_112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEybPNS_16OperationContextE+0x84) [0x10bdb94e4]

mongod(_ZN5mongo21WiredTigerRecordStore8IteratorC2ERKS0_PNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionEb+0x61) [0x10bdb5511]

mongod(_ZNK5mongo21WiredTigerRecordStore11getIteratorEPNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionE+0x6C) [0x10bdb32cc]

mongod(_ZN5mongo21WiredTigerRecordStoreC2EPNS_16OperationContextERKNS_10StringDataES5_bxxPNS_28CappedDocumentDeleteCallbackEPNS_20WiredTigerSizeStorerE+0x38A) [0x10bdaff6a]

mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextERKNS_10StringDataES5_RKNS_17CollectionOptionsE+0x134) [0x10bdac7e4]

mongod(ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKNSt3_112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEb+0x158) [0x10bd28168]

mongod(_ZN5mongo15KVStorageEngineC2EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x919) [0x10bd2afa9]

mongod(ZNK5mongo12_GLOBAL_N_117WiredTigerFactory6createERKNS_19StorageGlobalParamsERKNS_21StorageEngineLockFileE+0x11C) [0x10bdaa11c]

mongod(ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKNSt3_112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE+0x27C) [0x10bab88cc]

mongod(_ZN5mongoL14_initAndListenEi+0x328) [0x10b81a7c8]

mongod(_ZN5mongo13initAndListenEi+0x13) [0x10b819f53]

mongod(main+0x3BC) [0x10b81ee9c]

mongod(start+0x34) [0x10b819f34]

-----  END BACKTRACE  -----

2020-11-11T01:13:04.817+0200 I -        [initandlisten]

 

***aborting after invariant() failure

 

 

Any help would be be appreciated, thanks!



 Comments   
Comment by Edwin Zhou [ 17/Dec/20 ]

Hi alex_tsm@hotmail.com,

Thank you for providing the files. Unfortunately I wasn't able to repair this corruption, and it can be an issue with 3.0. Given that you're operating on an unsupported version of MongoDB, I advise that you upgrade to MongoDB 3.6 or later.

To avoid a problem like this in the future, it is our strong recommendation to:

Best,
Edwin

Comment by Alex TSM TSM [ 17/Dec/20 ]

Hello Edwin,

My sincere apology for not responding. I have uploaded the files and will respond faster in the future.

Thank you for all the effort!

Br,
Alexander T.

Comment by Edwin Zhou [ 17/Dec/20 ]

Hi alex_tsm@hotmail.com,

Would you like to continue with the metadata repair? If this is still an issue for you, would you please upload wiredTiger.wt and wiredTiger.turtle to the secure upload portal?

Thanks,
Edwin

Comment by Edwin Zhou [ 02/Dec/20 ]

Hi alex_tsm@hotmail.com,

I appreciate your concern. The files I requested only contain metadata about other *.wt files but not the data stored in the database. Collection and index data is stored in collection-*.wt and index-*.wt files respectively, which I have not requested.

Best,
Edwin

Comment by Alex TSM TSM [ 02/Dec/20 ]

Hello,

First of all thank you for the help, very much appreciated. Is any of the information stored in the database exposed through the requested files?

 

Best regards,

Alex

Comment by Edwin Zhou [ 02/Dec/20 ]

Hi alex_tsm@hotmail.com,

As MongoDB 3.0.1 is EOL, we can offer a one time effort to help and try to repair the .wt files. Please attach copies of the wiredTiger.wt and wiredTiger.turtle files to the upload portal and we can attempt a metadata-only repair effort using internal tools. Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Keep in mind that this repair effort may not be successful, and that diagnosing corruption issues requires significant information and effort.

Kind regards,
Edwin

Comment by Alex TSM TSM [ 28/Nov/20 ]

Logs for start: 

_____________________

sudo mongod -storageEngine wiredTiger --directoryperdb --dbpath /Users/alex_fimm_dev/mongodb/data --setParameter failIndexKeyTooLong=false

_____________________

 

2020-11-28T14:21:39.615+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),

2020-11-28T14:21:40.124+0200 I CONTROL  [initandlisten] MongoDB starting : pid=53363 port=27017 dbpath=/Users/alex/mongodb/data 64-bit host=lm50G3QD

2020-11-28T14:21:40.124+0200 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.

2020-11-28T14:21:40.124+0200 I CONTROL  [initandlisten]

2020-11-28T14:21:40.124+0200 I CONTROL  [initandlisten] db version v3.0.1

2020-11-28T14:21:40.124+0200 I CONTROL  [initandlisten] git version: 534b5a3f9d10f00cd27737fbcd951032248b5952

2020-11-28T14:21:40.124+0200 I CONTROL  [initandlisten] build info: Darwin bs-osx108-7 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49

2020-11-28T14:21:40.124+0200 I CONTROL  [initandlisten] allocator: system

2020-11-28T14:21:40.124+0200 I CONTROL  [initandlisten] options: { setParameter:

{ failIndexKeyTooLong: "false" }

, storage: { dbPath: "/Users/alex/mongodb/data", directoryPerDB: true, engine: "wiredTiger" } }

2020-11-28T14:21:40.138+0200 I NETWORK  [initandlisten] waiting for connections on port 27017

 

_____________________

 

How can I get the logs from repair?

 

Is there no way of restoring the database form the folder containing the .wt files ?

 

Thanks again for the help and sorry for late reply

 

Comment by Edwin Zhou [ 18/Nov/20 ]

Hi alex_tsm@hotmail.com,

Thanks for informing us of the steps you took. Could you attach the following:

  • The logs of the repair operation.
  • The logs of any attempt to start mongod after the repair operation completed.

Best,

Edwin

Comment by Alex TSM TSM [ 18/Nov/20 ]

Hello and thanks for your response.

I ran mongod with --repair flag successfully and then fired mongod normally successfully as well. Problem now is that when I launch  'mongo' in another terminal and run the 'show dbs' not all show up and they all show 0 mb in size. When I switch to the database that I am interested in (which does appear but size 0 as well) and run the 'show collections' command they are all missing. When I check the data directory the file with the folder with the database name is there and all .wt files are there and taking a total space of 7gb. Is there any diagnosis command I can run to repair or create a new database using the existing broken database . I do not have a dump... Any advice?

Comment by Edwin Zhou [ 13/Nov/20 ]

Hi alex_tsm@hotmail.com,

Can you validate your ulimit settings using ulimit -a and checking them against the recommended ulimit settings for 3.0?

Best,

Edwin

Comment by Alex TSM TSM [ 10/Nov/20 ]

Using MongoDB shell version: 3.0.1

Generated at Thu Feb 08 05:28:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.