[SERVER-9818] Problems when starting MongoDB with > 1023 FDs Created: 30/May/13 Updated: 11/Jul/16 Resolved: 10/Sep/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 2.4.3, 2.5.0 |
| Fix Version/s: | 2.5.3 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Oliver John | Assignee: | Matt Dannenberg |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
RHEL 6.3 x86_64 |
||
| Attachments: |
|
||||||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||
| Operating System: | Linux | ||||||||||||||||||||||||||||
| Steps To Reproduce: | Start mongodb instance with more than 1024 datafiles used for the local db. |
||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Description |
|
When running a 3 node replSet with lots of data and currently and have more than 1500 datafiles, you got the following error messages at restart. This bug might correspond to https://jira.mongodb.org/browse/SERVER-8515 Version 2.4.3 Thu May 30 15:57:27.912 [initandlisten] db version v2.4.3 Thu May 30 15:57:27.937 [initandlisten] journal dir=/data/mongo-longterm/journal -------------------------------- Version 2.5.1-pre from github Thu May 30 15:58:30.605 [initandlisten] db version v2.5.1-pre- Thu May 30 15:58:30.633 [initandlisten] journal dir=/data/mongo-longterm/journal |
| Comments |
| Comment by auto [ 19/Sep/13 ] | |||||
|
Author: {u'username': u'dannenberg', u'name': u'Matt Dannenberg', u'email': u'matt.dannenberg@10gen.com'}Message: | |||||
| Comment by auto [ 10/Sep/13 ] | |||||
|
Author: {u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}Message: | |||||
| Comment by auto [ 04/Sep/13 ] | |||||
|
Author: {u'username': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}Message: Revert " This reverts commit 607ae1384b26ebcb5e275a845f9806104852257f. | |||||
| Comment by auto [ 04/Sep/13 ] | |||||
|
Author: {u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}Message: | |||||
| Comment by Shaun Verch [ 10/Jun/13 ] | |||||
|
Hi Oliver, Even in standalone mode I believe that mongod will still try to open all these local database files. If that procedure does work for you, let us know, and if doesn't, also let us know and we can walk you through the process of using mongodump and mongorestore. Thanks! | |||||
| Comment by Oliver John [ 10/Jun/13 ] | |||||
|
Hi Shaun, First of all, thanks for your support! We're currently decreasing the oplogSize and try to start over with a fresh local-db as described in http://docs.mongodb.org/manual/tutorial/change-oplog-size/. This should fix our problem for now, i think. Thanks again and cheers, | |||||
| Comment by Shaun Verch [ 07/Jun/13 ] | |||||
|
Hi Tim, By default, MongoDB preallocates the oplog as a function of disk size. You can change this using the --oplogSize option to mongod. See http://docs.mongodb.org/manual/reference/configuration-options/#oplogSize for more details on this. This unreasonable default for the oplog size will be fixed as Have you inserted any data yet? Can you just wipe the data directories and start over? If not, let us know and we can walk you through the dump restore process. Thanks, | |||||
| Comment by Tim Eggert [ 07/Jun/13 ] | |||||
|
Hi Shaun, thank you very much for your detailed answer. Is it possible to circumvent this behaviour? Thanks and kind regards, Tim | |||||
| Comment by Shaun Verch [ 07/Jun/13 ] | |||||
|
Hi Oliver, It appears that the database that has many data files for your system is the local database. While you can store information here, mongodb also uses the local database for metadata and handles it differently from other databases. What are you storing in the local database? One thing you can do to get around this issue is to run mongodump in "direct client" mode. This will interact with the data files directly without actually starting up the server:
Then, you can restore all your data into a different database using the mongorestore command. With a mongod instance running:
Without a mongod instance running:
Quick example:
You can also adapt this to dump/restore the entire local database. The deeper issue here will be solved when our call to select is replaced as part of Thanks, | |||||
| Comment by Oliver John [ 06/Jun/13 ] | |||||
|
I attached a logfile with the full verbose output (-vvvvv). Thanks, | |||||
| Comment by Shaun Verch [ 04/Jun/13 ] | |||||
|
Hi Oliver and Tim, We are having some issues reproducing the problem in this ticket. Could you try running mongod with the -vvvvv option and post the output? That would give us more information about what files the database is opening on startup. Thanks! | |||||
| Comment by Shaun Verch [ 03/Jun/13 ] | |||||
|
Hi Tim, We are working on a reproduction of the issue. We'll post updates as we have them. In the meantime, it would be helpful if you also posted some details about your environment. In particular: 1. Number of data files you have Thanks, | |||||
| Comment by Tim Eggert [ 03/Jun/13 ] | |||||
|
Hi, could you please give an update on the current ticket status. We are also experiencing the same issue in our production environments. Thank you very much and kind regards, Tim Eggert |