[SERVER-11736] Database crashes when there is insufficient rights to create temporary _tmp directory for index creation. Created: 15/Nov/13  Updated: 10/Dec/14  Resolved: 09/Dec/13

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Aleksei Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: Linux
Steps To Reproduce:

Launch mongo cluster under mongo user. Some db files will be created in data dir.
Change owner of data dirs (but not owner of files in it) to root on all mongo nodes.
Create any new index for db.
As a result all nodes will crash at same time since they cannot create _tmp dir in data dir.

Participants:

 Description   

Database crashes when there is insufficient rights to create temporary _tmp directory for index creation.



 Comments   
Comment by Daniel Pasette (Inactive) [ 09/Dec/13 ]

Thanks for the report.
In 2.4.5, mongod would shut down if it is unable to create a file it needs as in the following log snippet:

Mon Dec  9 03:07:42.398 [initandlisten] connection accepted from 127.0.0.1:59702 #1 (1 connection now open)
Mon Dec  9 03:07:47.459 [conn1] build index test.f { x: 1.0 }
Mon Dec  9 03:07:47.459 [conn1] ERROR: Uncaught std::exception: boost::filesystem::create_directory: Permission denied: "/data/crash/_tmp", terminating
Mon Dec  9 03:07:47.459 dbexit:
Mon Dec  9 03:07:47.459 [conn1] shutdown: going to close listening sockets...
Mon Dec  9 03:07:47.459 [conn1] closing listening socket: 9
Mon Dec  9 03:07:47.459 [initandlisten] now exiting
Mon Dec  9 03:07:47.459 [conn1] closing listening socket: 10
Mon Dec  9 03:07:47.459 dbexit: ; exiting immediately

In 2.5.4, it will continue to try and create the file every 10 secs, which allows the db to stay running and available for reads while the permissions are fixed.

Comment by Aleksei [ 06/Dec/13 ]

If I remember correctly the version was 2.4.5. (2.4.x for sure)

Comment by Daniel Pasette (Inactive) [ 06/Dec/13 ]

What version of the server are you using? I tried to quickly reproduce in the latest code and got the following error:

2013-12-06T01:28:03.604-0500 [conn1] build index on: test.f properties: { v: 1, key: { x: 1.0 }, name: "x_1", ns: "test.f" }
2013-12-06T01:28:05.563-0500 [FileAllocator] allocating new datafile /data/crash/test.1, filling with zeroes...
2013-12-06T01:28:05.563-0500 [FileAllocator] creating directory /data/crash/_tmp
2013-12-06T01:28:05.563-0500 [FileAllocator] error: failed to allocate new file: /data/crash/test.1 size: 134217728 boost::filesystem::create_directory: Permission denied: "/data/crash/_tmp".  will try again in 10 seconds
2013-12-06T01:28:15.561-0500 [conn1] Assertion: 12520:new file allocation failure

Generated at Thu Feb 08 03:26:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.