[SERVER-5670] externally preallocated data files may be ignored when allocating extents Created: 20/Apr/12  Updated: 11/Jul/16  Resolved: 07/Jun/12

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 2.1.2

Type: Bug Priority: Major - P3
Reporter: Aaron Staple Assignee: Eric Milkie
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-5976 new database files might not always b... Closed
Operating System: ALL
Participants:

 Description   

In the following scenario externally preallocated data files may be skipped:

1) data files (test.0,1,2,...) are allocated manually (for example with dd).
2) Mongod is started and database 'test' is created to use these data files.
3) Mongod is restarted, and the 'test' database is opened. Because the database already exists at the time it is opened, openAllFiles() is called which increments the database's 'n' to the total number of existing data files minus one.
4) A collection in the test database grows in size, with growing extent sizes, until the followupSize() call to determine a new collection size returns Extent::maxSize().
5) Database::suitableFile() is called with Extent::maxSize(). No data file under the Database's 'n' limit can fit an extent of this size because the specified max extent size is equal to the max file size and some room is needed for the file header.
6) Database::suitableFile() fails to identify an existing file (under the 'n' limit) that can fit an extent of Extent::maxSize(), so it allocates a new file with addAFile() and returns that new file.
7) Subsequent extent allocations also request a file for an extent of size Extent::maxSize() and skip the unused files as well.

I did this to demonstrate the issue:

I manually preallocated some data files

for i in `seq 0 29`; do dd if=/dev/zero of=test.$i bs=8192 count=262016 ; done

I started mongod and began running this script

db.createCollection( 'c', { autoIndexId:false } );
big = new Array( 1000000 ).toString();
while( 1 ) {
    db.c.insert( { b:big } );
    db.getLastError();
}

I let the script run for a little while, then stopped and restarted mongod and ran the script again. Most of the data files that were unused by the prior mongod session remained unused as new data files were allocated.



 Comments   
Comment by auto [ 01/Jun/12 ]

Author:

{u'login': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: removing invalid code that breaks SERVER-5670
Branch: master
https://github.com/mongodb/mongo/commit/bc4b27c7a84e1172dc32809603c425abf9e97050

Comment by Aaron Staple [ 20/Apr/12 ]

@eric There are multiple ways of fixing this specific issue. For example we could alternatively modify openExistingFiles() – and in fact it seems this has already been done in 2.1 and the issue may have been fixed there as a result. It also makes sense to me that the size checks for existing and new files should be consistent.

Comment by Thorn Roby [ 20/Apr/12 ]

I was about to rebuild our current production server using the new architecture described in CS-2838, which involves preallocating a large set of 2GB datafiles. I have been using the file size that matches datafiles created by mongod normally (2146435072), using dd with a single block of this size from /dev/zero (I settled on this after having some problems I don't remember exactly, when I used a file size computed as an exact 2GiB, which is slightly larger than this number). Does the scenario described here imply that I should use a slightly larger (maybe 8192) size to allow room for the file header? Or should I just hold off until you do some more investigation of the issue? Currently our production server is running well, as is the process of copying the data into the new server which is to become the replacement production server, so I'm under no particular pressure to go ahead with the cutover if it is likely that the datafile preallocation problem might be better understood sometime in the next week or so.

Comment by Eric Milkie [ 20/Apr/12 ]

So from what you're saying, it sounds like the issue is that the Extent max size should not be the same as the max file size, due to the extra space taken up by a file header?
Or rather, that the check for extent space in a new file is different than the check for space in an existing file (and to fix this, we should make them the same).

Generated at Thu Feb 08 03:09:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.