[SERVER-5194] Windows version of mongod should manage its own working set Created: 04/Mar/12  Updated: 06/Dec/22  Resolved: 14/Sep/18

Status: Closed
Project: Core Server
Component/s: MMAPv1, Performance, Stability, Storage, Usability
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tad Marshall Assignee: Backlog - Storage Execution Team
Resolution: Won't Fix Votes: 6
Labels: Windows, performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows, especially with journaling and heavy insert load


Issue Links:
Depends
Duplicate
is duplicated by SERVER-4300 "Assertion: 10000:out of memory BufBu... Closed
is duplicated by SERVER-4478 Mongod should allow a setting to peri... Closed
is duplicated by SERVER-4301 Why does Mongo become slower when RAM... Closed
Related
related to SERVER-3911 memory leak with journaling in windows Closed
related to SERVER-538 SetProcessWorkingSetSizeEx for win32? Closed
Assigned Teams:
Storage Execution
Operating System: Windows
Participants:

 Description   

The Windows operating system does not do a good job of maintaining an optimal working set (pages assigned to physical memory) for mongod.exe. From observation, it appears that memory mapped files are given too high a priority and may be allowed to consume so much memory that system responsiveness drops and out-of-memory failures become possible. Linux behaves differently, but on Windows we need to take a more active role in managing our footprint.

On newer versions of Windows (Windows Server 2003 and Windows Server 2003 R2, Vista, Windows 7, Windows Server 2008 and Windows Server 2008 R2) we can use a Windows API (SetProcessWorkingSetSizeEx()) to cap our memory consumption at a level that we determine will work well on the hardware we find ourselves running on. In older versions of Windows (Windows XP) we can use EmptyWorkingSet() or SetProcessWorkingSetSize(handle, -1, -1) to free the entire working set when we approach our predetermined limit. Because of the way Windows first moves pages onto a standby list, these pages can be brought back into the working set quickly ("soft" page faults), so freeing the entire working set is not as bad as you might expect. Also, Windows does a poor enough job of keeping the right pages in the working set that throwing it all away and building up a new working set may be better anyway.

We could also look at tracking the pages that we absolutely know we want in memory and empty the working set even on newer versions of Windows and then fault all of the pages we want back into memory on a worker thread (by reading one byte from each page we want). Some performance testing will be required to see if this is a workable strategy or not.


Generated at Thu Feb 08 03:08:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.