[SERVER-24776] Unable to recover after fassert(): too many open files Created: 24/Jun/16 Updated: 14/Jul/16 Resolved: 24/Jun/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | zeerak waseem | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Steps To Reproduce: | 1) Write to database, get 'too many files' exception |
| Participants: |
| Description |
|
Hey guys, My databases and all collections just disappeared after a crash. I've described the issue here: http://stackoverflow.com/questions/38004375/mongodb-databases-disappeared?noredirect=1#comment63461612_38004375 Please let me know whatever you need and I'll make sure you get it. It should be noted last I checked one database was 5GB and the other was < 500 MB. |
| Comments |
| Comment by zeerak waseem [ 25/Jun/16 ] | |||||||||||||||||||||||||||
|
Hey, so turns out you weren't quite right. Changing ulimit (250 -> 1000) was the step that did not allow me to restart the daemon afterwards. [/data] mongod What fixed that was to move the journals out of the database path which allows me to restart the daemon. Moving it back and restarting the daemon allows me to start it, but it still does not produce any of the records I had previously put into it. Primarily saying this for completeness. I'll post it to the mailing list once I have time. I'll try updating to latest patch, but not hoping for much there. | |||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 24/Jun/16 ] | |||||||||||||||||||||||||||
|
Please see the documentation for recommended ulimit settings, that should do the trick. | |||||||||||||||||||||||||||
| Comment by zeerak waseem [ 24/Jun/16 ] | |||||||||||||||||||||||||||
|
Hey Ramon, thanks for the quick answer. I I'll post it to the user group as well but (since my experiments for my thesis are to be done in a couple of days I'll ask here too) I tried to increase the max number of files which was the change that made me unable to start the daemon as well. | |||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 24/Jun/16 ] | |||||||||||||||||||||||||||
|
Adding the beginning of the log and the fatal assertion part for convenience:
zeerakw, looks like mongod shut itself down because of insufficient resources, and the error every time you restart is the same, so you'll need to increase your limits. Also I see you're using 3.2.0 – I'd recommend you always use the latest patch release in a given version (3.2.7 at the moment). Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources. Regards, | |||||||||||||||||||||||||||
| Comment by zeerak waseem [ 24/Jun/16 ] | |||||||||||||||||||||||||||
|
Scripts that caused the conflict. Specifically look at ``write_all_user_tweets`` in linguistic_feats.py |