[DOCS-4] Recipe for pre-allocation of oplog produces inefficiency Created: 20/Apr/11  Updated: 02/Feb/12  Resolved: 02/Feb/12

Status: Closed
Project: Documentation
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Jason R. Coombs Assignee: Kristina Chodorow (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:
Days since reply: 12 years, 2 weeks, 6 days ago

 Description   

The manual page titled "Halted Replication" gives directions to manually pre-allocate the oplog files using a shell script. In my experience, following these directions produces 2GB of wasted space.

I used the script to produce 13 2GB files, then started mongod using --oplogSize 26000, but then mongod created another 2GB file, consuming 28GB for a 26GB oplog.

It would be nice if the recipe could be updated to correctly pre-generate the oplog files as mongodb would do if it generated them itself. Even better would be to add a feature to mongod to generate files for an empty oplog. e.g. mongod --generateOplogOnly --master --oplogSize 26000 --dbpath /data/prealloc would generate the files and then just exit. It may be possible to do this now (by leaving off generateOplogOnly), but my worry is that the generated files may have metadata in them (in particular the oplog.$main collection) which doesn't match the recommended procedure in the docs.

Ultimately, the docs should reflect a procedure that allows for offline generation but ultimately produces the same outcome as if the server had been started with no oplog.



 Comments   
Comment by Kristina Chodorow (Inactive) [ 02/Feb/12 ]

Sorry, missed that question.

The recipe will only generate local.ns. I've modified the page to add instructions to delete local.ns before moving the files.

Comment by Jason R. Coombs [ 02/Feb/12 ]

My question remains. The recipe using mongodb to generate the oplogs will generate files in /tmp/local besides just local.*. I suspect if one tries to follow the steps, it will fail when it tries to move the *.ns files over the production *.ns files. Furthermore, if future versions of mongodb generate additional files in that directory, they will overwrite or fail. Probably the easiest fix is to update step 3 to read:

{{{
Shut down the mongod master (kill -2) and then replace the oplog files:

$ mv /data/db/local.* /safe/place
$ mv /tmp/local/local.* /data/db/
}}}

Comment by Sam Kleinman (Inactive) [ 02/Feb/12 ]

It looks like this is resolved?

Kristina, feel free to close, or assign back to me if more work is needed on this.

Comment by Jason R. Coombs [ 25/Aug/11 ]

You're right. This appears to be resolved. I attempted to replicate my finding using MongoDB 1.8.1 and it behaves as expected (using the pre-allocated log files and not allocating additional).

Thanks Kristina for making the change to the docs. One question, though: should the instructions instruct to delete the *.ns files before copying to the production directory?

Comment by Kristina Chodorow (Inactive) [ 25/Aug/11 ]

I can't reproduce, what version of the server is this on?

I get:

Thu Aug 25 17:23:43 [initandlisten] ******
Thu Aug 25 17:23:43 [initandlisten] creating replication oplog of size: 26000MB...
Thu Aug 25 17:23:43 [FileAllocator] allocating new datafile /dbpath/local.ns, filling with zeroes...
Thu Aug 25 17:23:43 [FileAllocator] done allocating datafile /dbpath/local.ns, size: 16MB, took 0 secs
Thu Aug 25 17:23:43 [initandlisten] ******

Doing an ls on the dbpath showed that it did not allocate any extra files.

You can start with --oplogSize=whatever --dbpath=/tmppath --port=some-other-port to prealloc, I'll add that to the docs.

Generated at Thu Feb 08 07:37:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.