[SERVER-9271] Failed to allocate new file on a database over mounted folder on Windows Created: 06/Apr/13  Updated: 16/Nov/21  Resolved: 03/Nov/16

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.4.1
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: David Verdejo Assignee: Daniel Gottlieb (Inactive)
Resolution: Done Votes: 0
Labels: allocate, error, folder, mounted, space, windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Fri Apr 05 18:44:12.308 [initandlisten] db version v2.4.1
Fri Apr 05 18:44:12.308 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49


Attachments: PNG File mongodb_allocate_new_file.PNG     PNG File mongodb_replica_fail.PNG    
Issue Links:
Related
related to SERVER-9264 Repair process fails with a mount fol... Closed
Backwards Compatibility: Fully Compatible
Operating System: Windows
Steps To Reproduce:

1) Create a database
2) Move the database to a mounted folder
3) Insert new data until new space is needed

Sprint: Storage 2016-11-21
Participants:

 Description   

We create a database over mounted folder. When the database needs to allocate more space, there is a error message on the log file:

Sat Apr 06 17:07:52.100 [FileAllocator] mongod.exe    ...\src\mongo\util\stacktrace.cpp(189)                           mongo::printStackTrace+0x3e
Sat Apr 06 17:07:52.100 [FileAllocator] mongod.exe    ...\src\mongo\util\assert_util.cpp(114)                          mongo::verifyFailed+0xdc
Sat Apr 06 17:07:52.100 [FileAllocator] mongod.exe    ...\src\mongo\util\file_allocator.cpp(73)                        mongo::ensureParentDirCreated+0x121
Sat Apr 06 17:07:52.100 [FileAllocator] mongod.exe    ...\src\mongo\util\file_allocator.cpp(283)                       mongo::FileAllocator::run+0x342
Sat Apr 06 17:07:52.100 [FileAllocator] mongod.exe    ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(180)  boost::`anonymous namespace'::thread_start_function+0x21
Sat Apr 06 17:07:52.100 [FileAllocator] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)      _callthreadstartex+0x17
Sat Apr 06 17:07:52.100 [FileAllocator] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(292)      _threadstartex+0x7f
Sat Apr 06 17:07:52.100 [FileAllocator] KERNEL32.DLL                                                                   BaseThreadInitThunk+0x1a
Sat Apr 06 17:07:52.100 [FileAllocator] error: failed to allocate new file: E:\mongodb\data\db\cacheagg\cacheagg.6 size: 2146435072 assertion src\mongo\util\file_allocator.cpp:73.  will try again in 10 seconds

After that, the replica set fails:

{
                        "_id" : 3,
                        "name" : "log-mng21:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 80796,
                        "optime" : Timestamp(1365260435000, 113),
                        "optimeDate" : ISODate("2013-04-06T15:00:35Z"),
                        "lastHeartbeat" : ISODate("2013-04-06T15:13:31Z"),
                        "pingMs" : 0,
                        "errmsg" : "syncThread: 14031 Can't take a write lock while out of disk space"
}

Yesterday, I opened a ticket (SERVER-9264) related with problems with repair over mounted folder. I received a workaround to solve the problem.



 Comments   
Comment by Daniel Gottlieb (Inactive) [ 03/Nov/16 ]

MongoDB 3.2 upgraded to Boost 1.56. I've confirmed the upgrade fixes this problem.

Comment by Ramon Fernandez Marina [ 23/Jan/15 ]

Thanks for the detailed information ogurets, we're adding this ticket to the 3.1 planning round.

Comment by Daniel Pasette (Inactive) [ 26/Dec/14 ]

Thanks for the detailed repro and explanation. We'll take a look soon.
Dan

Comment by Ogurets Johnson [ 24/Dec/14 ]

Hi! I was able to reproduce this:
Windows 2003 Server, Mongodb build win32-x86_64-2.6.4, mongod service startup arguments (no config file):

--dbpath c:\databases\mongo --smallfiles --directoryperdb --service

Inside c:\databases\mongo I've created a junction named "junction" (this is a major point of this issue!) pointing to the original place where database files are stored. And then, when database fills up a little (this is a second important condition), bam!:

2014-12-24T17:00:59.484+0300 [FileAllocator] allocating new datafile c:\databases\mongo\junction\junction.1, filling with zeroes...
2014-12-24T17:00:59.484+0300 [FileAllocator] Assertion failure boost::filesystem::is_directory(parent) src\mongo\util\file_allocator.cpp 79
2014-12-24T17:01:00.109+0300 [FileAllocator] ...\src\mongo\util\stacktrace.cpp(169) mongo::printStackTrace+0x43
2014-12-24T17:01:00.109+0300 [FileAllocator] ...\src\mongo\util\log.cpp(127) mongo::logContext+0x9c
2014-12-24T17:01:00.109+0300 [FileAllocator] ...\src\mongo\util\assert_util.cpp(115) mongo::verifyFailed+0x14a
2014-12-24T17:01:00.109+0300 [FileAllocator] ...\src\mongo\util\file_allocator.cpp(79) mongo::ensureParentDirCreated+0x158
2014-12-24T17:01:00.109+0300 [FileAllocator] ...\src\mongo\util\file_allocator.cpp(307) mongo::FileAllocator::run+0x39a
2014-12-24T17:01:00.109+0300 [FileAllocator] ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185) boost::`anonymous namespace'::thread_start_function+0x21
2014-12-24T17:01:00.109+0300 [FileAllocator] f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314) _callthreadstartex+0x17
2014-12-24T17:01:00.109+0300 [FileAllocator] f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(292) _threadstartex+0x7f

Short explanation: boost::filesystem::is_directory function doesn't recognize Windows junctions as directories and returns false (this issue even has a ticket - https://svn.boost.org/trac/boost/ticket/9016).

The cure would be not to use is_directory at all - try to create a file, catch an error. Older Mongo versions seem to do exactly that, but this additional check was added for some reason, so I cannot just patch it out before consulting a person who placed it there.

Comment by Stennie Steneker (Inactive) [ 19/Mar/14 ]

Hi David,

Please be advised I'm now closing this issue as we have been unable to reproduce the reported problem.

If you do have any further information that would help us reproduce this issue (as per the last questions from Luke), please let us know.

Thanks,
Stephen

Comment by Luke Lovett [ 06/Nov/13 ]

Hi David,

I tried to replicate your setup & steps on our end. I'm using Windows 2008 Server R2 with SP1. This is what I did:

1. Started a replica set with a dbpath located somewhere on the C drive and inserted some data
2. Created a virtual disk image with limited space (100GB) and mounted it to C:\Disks\mnt
3. Moved the data\db directory from the C drive to the new disk image mounted on C:\Disks\mnt
4. Re-started the replica set pointed at the new dbpath and inserted even more data until new data files needed to be allocated

In the end, everything worked fine for me with MongoDB 2.4.1.

Are you still experiencing this problem? If so, can I ask how you created your disk image and how you mounted it? Are you using NTFS as your file system or something else? What flags are you using when you run mongod?

Thanks,
Luke

Comment by David Verdejo [ 11/Apr/13 ]

I tried to change from mounted folder to symbolic lync but this fails too.

I downgraded to version 2.2.4 and everything works fine.

Comment by David Verdejo [ 08/Apr/13 ]

Folder structure
================

E:\mongodb_rscacheagg\data\db--> DBPATH
E:\mongodb_rscacheagg\data\db\cacheagg --> MOUNTED FOLDER

Content
=======

Volume in drive E has no label.
Volume Serial Number is 5C20-0930

Directory of E:\mongodb_rscacheagg\data\db

07/04/2013 09:32 <DIR> .
07/04/2013 09:32 <DIR> ..
06/04/2013 17:42 <DIR> admin
06/04/2013 17:55 <JUNCTION> cacheagg [??\Volume

{34022341-8ff0-11e2-93f7-005056a032fd}

] --> MOUNTED FOLDER
07/04/2013 09:49 <DIR> cachehoteles
07/04/2013 22:26 <DIR> journal
06/04/2013 17:29 <DIR> local
07/04/2013 08:05 5 mongod.lock
07/04/2013 09:59 <DIR> _tmp
1 File(s) 5 bytes

Directory of E:\mongodb_rscacheagg\data\db\admin

06/04/2013 17:42 <DIR> .
06/04/2013 17:42 <DIR> ..
06/04/2013 17:42 67.108.864 admin.0
06/04/2013 17:42 134.217.728 admin.1
06/04/2013 17:42 16.777.216 admin.ns
3 File(s) 218.103.808 bytes

Directory of E:\mongodb_rscacheagg\data\db\cacheagg

06/04/2013 17:42 67.108.864 cacheagg.0
06/04/2013 17:42 134.217.728 cacheagg.1
06/04/2013 17:42 268.435.456 cacheagg.2
06/04/2013 17:42 536.870.912 cacheagg.3
06/04/2013 17:43 1.073.741.824 cacheagg.4
06/04/2013 17:43 2.146.435.072 cacheagg.5
07/04/2013 07:13 2.146.435.072 cacheagg.6
06/04/2013 17:42 16.777.216 cacheagg.ns
07/04/2013 07:13 <DIR> _tmp
8 File(s) 6.390.022.144 bytes

Directory of E:\mongodb_rscacheagg\data\db\cacheagg_tmp

07/04/2013 07:13 <DIR> .
07/04/2013 07:13 <DIR> ..
0 File(s) 0 bytes

Directory of E:\mongodb_rscacheagg\data\db\journal

07/04/2013 22:26 <DIR> .
07/04/2013 22:26 <DIR> ..
07/04/2013 22:26 1.073.750.016 j._7
07/04/2013 22:26 0 j._8
08/04/2013 07:21 88 lsn
3 File(s) 1.073.750.104 bytes

Directory of E:\mongodb_rscacheagg\data\db\local

06/04/2013 17:29 <DIR> .
06/04/2013 17:29 <DIR> ..
06/04/2013 17:29 67.108.864 local.0
06/04/2013 17:29 134.217.728 local.1
06/04/2013 17:29 2.146.435.072 local.2
06/04/2013 17:29 2.146.435.072 local.3
06/04/2013 17:29 2.146.435.072 local.4
06/04/2013 17:29 16.777.216 local.ns
06/04/2013 17:29 <DIR> _tmp
6 File(s) 6.657.409.024 bytes

Directory of E:\mongodb_rscacheagg\data\db\local_tmp

06/04/2013 17:29 <DIR> .
06/04/2013 17:29 <DIR> ..
0 File(s) 0 bytes

Directory of E:\mongodb_rscacheagg\data\db_tmp

07/04/2013 09:59 <DIR> .
07/04/2013 09:59 <DIR> ..
0 File(s) 0 bytes

PERMISSIONS
===========

  • Parent folder

C:\Users\Administrator>ICACLS e:\mongodb_rscacheagg\data\db
e:\mongodb_rscacheagg\data\db BUILTIN\Administrators:(F)
BUILTIN\Administrators:(I)(OI)(CI)(F)
NT AUTHORITY\SYSTEM:(I)(OI)(CI)(F)
CREATOR OWNER:(I)(OI)(CI)(IO)(F)
BUILTIN\Users:(I)(OI)(CI)(RX)
BUILTIN\Users:(I)(CI)(AD)
BUILTIN\Users:(I)(CI)(WD)

  • Mounted folder --> inheritance from parent folder

C:\Users\Administrator>ICACLS e:\mongodb_rscacheagg\data\db\cacheagg
e:\mongodb_rscacheagg\data\db\cacheagg BUILTIN\Administrators:(F)
BUILTIN\Administrators:(I)(OI)(CI)(F)
NT AUTHORITY\SYSTEM:(I)(OI)(CI)(F)
CREATOR OWNER:(I)(OI)(CI)(IO)(F)
BUILTIN\Users:(I)(OI)(CI)(RX)
BUILTIN\Users:(I)(CI)(AD)
BUILTIN\Users:(I)(CI)(WD)

Comment by Eliot Horowitz (Inactive) [ 07/Apr/13 ]

What kind of mount is this?
Can you send the contents of that directory, the parent, permissions, etc...

Comment by David Verdejo [ 07/Apr/13 ]

No, the mounted folder has 100 GB of free space. The problema is with allocation of free space.

I made a test: I downgraded to version 2.2.4 and everything works fine.

Comment by Eliot Horowitz (Inactive) [ 07/Apr/13 ]

Note:

{ "_id" : 3, "name" : "log-mng21:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 80796, "optime" : Timestamp(1365260435000, 113), "optimeDate" : ISODate("2013-04-06T15:00:35Z"), "lastHeartbeat" : ISODate("2013-04-06T15:13:31Z"), "pingMs" : 0, "errmsg" : "syncThread: 14031 Can't take a write lock while out of disk space" }

Looks like you are out of disk space?

Generated at Thu Feb 08 03:19:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.