The Windows operating system does not do a good job of maintaining an optimal working set (pages assigned to physical memory) for mongod.exe. From observation, it appears that memory mapped files are given too high a priority and may be allowed to consume so much memory that system responsiveness drops and out-of-memory failures become possible. Linux behaves differently, but on Windows we need to take a more active role in managing our footprint.

On newer versions of Windows (Windows Server 2003 and Windows Server 2003 R2, Vista, Windows 7, Windows Server 2008 and Windows Server 2008 R2) we can use a Windows API (SetProcessWorkingSetSizeEx()) to cap our memory consumption at a level that we determine will work well on the hardware we find ourselves running on. In older versions of Windows (Windows XP) we can use EmptyWorkingSet() or SetProcessWorkingSetSize(handle, -1, -1) to free the entire working set when we approach our predetermined limit. Because of the way Windows first moves pages onto a standby list, these pages can be brought back into the working set quickly ("soft" page faults), so freeing the entire working set is not as bad as you might expect. Also, Windows does a poor enough job of keeping the right pages in the working set that throwing it all away and building up a new working set may be better anyway.

We could also look at tracking the pages that we absolutely know we want in memory and empty the working set even on newer versions of Windows and then fault all of the pages we want back into memory on a worker thread (by reading one byte from each page we want). Some performance testing will be required to see if this is a workable strategy or not.