[SERVER-22450] Mongodb sub-folders sometimes don't get u+x permission Created: 03/Feb/16 Updated: 18/Feb/16 Resolved: 18/Feb/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Admin, Storage, WiredTiger |
| Affects Version/s: | 3.2.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Radek | Assignee: | Kelsey Schubert |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | It's hard to reproduce... usually it happens only on slaves... We create a new database every day... and there were no problems since december... and then it could not create/replicate files from in db: ZZZZZZ-160130 (30.I.2016) at midnight. That means, on previous days, there were no problems with permissions. And now the same problem with ./diagnostic.data/ means – the issue still presists in 3.2.1... |
||||||||
| Participants: | |||||||||
| Description |
|
Same problem as described here: This time:
Context mongodb.log in attachement. Most important lines:
and then:
|
| Comments |
| Comment by Kelsey Schubert [ 18/Feb/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi radek, I'm glad that you have an effective workaround. We are still unable to reproduce this behavior: I followed the same upgrade path from MongoDB 3.0.3 to 3.2.1 and was able to write successfully to the diagnostics.data directory. The fact that this issue emerges only following system restarts could indicate that the mask of the mongod process is being incorrectly set. If you are able, you may want to examine the mask of a mongod exhibiting this issue. Or you may want to examine if there are any automated processes being executed on these nodes which could explain the change in permission rights between server restarts. The relevant code to create the diagnostic.data directory can be found here. We take the mask from the environment, so if there is an issue it may reside within the Boost.Filesystem library. For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group. Kind regards, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Radek [ 09/Feb/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
New/empty instance – whole log below (conf @ the end): // couldn't create ..../local.0
conf:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Radek [ 09/Feb/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
And now _tmp: ... It's crazy... (error opening file "/local_storage/mongodb/mongodb-wt-27017/_tmp/extsort.0": errno:13 Permission denied)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Radek [ 09/Feb/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello Thomas, 1. New databases are created (more-or-less) just after midnight (via insert command) $mdb = new MongoClient("mongodb://mdb4.XXXX.YY,mdb7.XXXX.YY,mdb9.XXXX.YY/?replicaSet=mainSet&readPreference=primary"); 1a. Problem(s) with permissions happens mostly (only? not sure) on slave nodes. 2. No, the problem is not limited to "diagnostic.data" folder ("diagnostic.data" folder is just a prove, that I do not controll the process of managing priorites of MongoDB folders. Since I create a new database every day – the problem with permissions to "signals-YYMMDD" databses' folders (on slaves) happens not deterministicaly, every (let's say) every 20-40+ days ? But if it happens – it stalls replications on given slave. 3. This issue occurs sporadicly on different affected nodes, on different clusters (and databses created also monthly, not daily). IIRC, it only happens on slaves, but I'm not sure on that. 4. Mongod(s) are restarted quite rarely, mostly on upgrades (once a month, once every two months). But as you can see from "diagnostic.data" directory – this issue happened just after a restart (after an version upgrade). A few words more:
But I'm happy to help in future. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 09/Feb/16 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi radek, I have a few questions to get a better idea of what is going on here.
Thank you, |