[SERVER-538] SetProcessWorkingSetSizeEx for win32? Created: 13/Jan/10  Updated: 03/Jan/18  Resolved: 18/Nov/15

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Eliot Horowitz (Inactive) Assignee: Unassigned
Resolution: Duplicate Votes: 6
Labels: Windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
duplicates SERVER-17861 Change the default storage engine to ... Closed
Related
is related to SERVER-5194 Windows version of mongod should mana... Closed
Backwards Compatibility: Fully Compatible
Participants:
Case:

 Description   

need to investigate



 Comments   
Comment by Daniel Pasette (Inactive) [ 18/Nov/15 ]

Resolving as a duplicate of SERVER-17861. Now that WiredTiger is the default storage engine, memory usage can be controlled by setting the WiredTiger cache size, though users must take into account additional memory usage requirements of the process and the specific workload.

Comment by AndrewK [ 10/Sep/14 ]

And again, anyone? Hello? Bueller? Bueller?

Seriously, how can this ticket still not be resolved? I've had an external process setting the max process size for the mongod job object since 2009 without issues.

Comment by AndrewK [ 28/May/13 ]

Ping... any sign of life in the progress of this ticket?

Comment by Li Tsunglin [ 01/Nov/12 ]

While SetProcessWorkingSetSize only works once when it is called, limitation set by job object will be applied always.

Comment by Tad Marshall [ 31/Oct/12 ]

Hi Li,

Thanks for the suggestion!

Did you try SetProcessWorkingSetSizeEx() and find that it didn't work for you?

My understanding of job objects was that they were for grouping related processes. I did not know that they provided features such as limiting working set sizes.

Does this have an advantage over the other API?

Tad

Comment by Li Tsunglin [ 31/Oct/12 ]

FYI, this is what we did for limiting working set usage on Windows 7.

#ifdef _WIN32
    if (params.count("workingset")) {
        int working_set = params["workingset"].as<int>();
        const __int64 oneMB = 1024*1024i64;
 
        ::HANDLE hJob = CreateJobObject(NULL, NULL);
			
        JOBOBJECT_EXTENDED_LIMIT_INFORMATION jobext;
        jobext.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_WORKINGSET | JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE;
        jobext.BasicLimitInformation.MaximumWorkingSetSize = working_set*oneMB;
        jobext.BasicLimitInformation.MinimumWorkingSetSize = oneMB;
        if (::SetInformationJobObject(hJob, JobObjectExtendedLimitInformation, &jobext, sizeof(jobext)) == FALSE) {
            out() << "failed to SetInformationJobObject, with maximum working set size " << working_set
                  << " MB, error = " << ::GetLastError() << endl;
        }
        else if (::AssignProcessToJobObject(hJob, ::GetCurrentProcess()) == FALSE) {
            out() << "failed to AssignProcessToJobObject, error = " << ::GetLastError() 
                  << ", this won't work under debugger." << endl;
        }
    }
#endif

Comment by Tad Marshall [ 04/Mar/12 ]

Consolidating closely related tickets into one.

Comment by Robert Wojciechowski [ 08/Dec/11 ]

I have since bumped it to 2GB (out of 4GB total). I have since lost the logs for this instance, but I can try to replicate in our development environment.

Comment by Tad Marshall [ 08/Dec/11 ]

I haven't done testing with WSRM (or SetProcessWorkingSetSizeEx) yet and this bug is marked for "2.1 planning" so it isn't getting attention yet, but if you have a log with anything interesting in it, could you add it to the bug, please?
A repair requires rewriting chunks of data, so a 1 GB working set limit might well be too low for it to work well. Do you need the limit to be so low? I suspect that the best value for working set size is some high fraction of your total RAM; the main objective should be to keep paging manageable by making sure that the working set is not larger than physical memory.

Comment by Robert Wojciechowski [ 08/Dec/11 ]

I have a potential wrench to throw out there. I'm using WSRM in 2008 R2 with an resource allocation policy allowing a maximum of 1GB to MongoDB, which has worked great for months. It does basically the same thing as SetProcessWorkingSizeEx just from the outside and requires Windows Server 2008 of course.

However, we recently had a server crash and when MongoDB came up it had to do a repair. Thing is, it wouldn't finish. I have no idea why, but I couldn't get MongoDB back up and running until I first removed the resource allocation policy in WSRM and tried again. That time it worked.

Is there something in the DB repair process that requires a certain amount of memory for a database of a certain size? Our database is about 11GB in size (16GB total for MongoDB including the journal, etc).

Comment by Tad Marshall [ 05/Dec/11 ]

Regarding my comment from Nov 30 2011 11:24:55 AM UTC where I suggested calling SetProcessWorkingSetSize() when SetProcessWorkingSetSizeEx() isn't available, this turns out to be useless. I watched a Mark Russinovitch talk where he specifically said that the maximum working set parameter of SetProcessWorkingSetSize() does nothing and hasn't for years. The Ex version, with the extra flag, does work, just the fallback for XP and Server 2003 won't help.

Comment by Tad Marshall [ 30/Nov/11 ]

Thanks, Andrew, that's great to know. Currently, this ticket is set at "2.1 planning", which means that I need to complete the stuff with "closer" fix-by versions before doing this, but I'm looking at Windows memory consumption and this might be helpful there. You'll see activity on this bug whenever I or someone else gets working on it; I wanted to get some of my thoughts written into the bug while I was reviewing it.

Comment by AndrewK [ 30/Nov/11 ]

Tad, for what it's worth, we have been using SetProcessWorkingSetSizeEx() in production since I logged this ticket (and then even before that). We've used it with almost every version of MongoDB since 1.2.4 under Window Server 2008 R2 and have never had a problem. It has worked perfectly - even where we have multiple MongoDB instances on the same machine.

I am more than happy to test your changes for you.

Comment by Tad Marshall [ 30/Nov/11 ]

We could use SetProcessWorkingSetSizeEx() on machines that have that function available (Vista, Server 2008, Windows 7, Server 2008 R2) and use SetProcessWorkingSetSize() for the rest (Windows XP, Server 2003). We would need to do some testing to see how much each of these helps, and document our findings on the Wiki. Also, these are Windows APIs that could be called at any time, so in addition to a command line switch we should implement a changeable setting (admin-only or localhost-only) that would let users "tune" a running instance, and document how and when to use that.

Comment by Ed Rooth [ 18/Apr/11 ]

Just found this blog post. Seems to solve the problem with an optional windows server native component.

http://www.captaincodeman.com/2011/02/27/limit-mongodb-memory-use-windows/

Comment by Robert Wojciechowski [ 06/Oct/10 ]

Please check out the following commit that allows you to set the limit and should support any Windows OS (errors out gracefully if you try to use it on an older version)

http://github.com/wojo/mongo/commit/a6e1b6892be9ea2733f4c2bc4059713dc0a7d459

I'm sure it needs some tiny tweaks (style, proper error codes, etc) but it works here in our environment well!

Comments?

Comment by Robert Wojciechowski [ 27/Jul/10 ]

Justin, absolutely. I'll do the same on my end. I just need to bite the bullet and get a mongo build environment up – shouldn't be too hard.

Comment by Justin Dearing [ 27/Jul/10 ]

Robert,

Agreed. I might try to implement it that way myself when I got some time.

However, rather than arguing to make our case to keep the ticket open, us community contributors should write a patch to dynamically load the function submit a pull request, and then the 10gen people can do their performance testing when we have code to show them.

Justin

Comment by Robert Wojciechowski [ 27/Jul/10 ]

Why was this implementing using a direct call? It could be done using LoadLibrary, GetProcAddress and a function pointer to dynamically load and then call the function SetProcessWorkingSetSizeEx depending on OS version. That way it'll be portable to all versions of Windows including XP.

This would be a very simple change and would satisfy all requirement I believe. If anyone needs help on this let me know. I really need to get a mongo build environment up anyway!

Can this be re-opened and fixed this way?

Comment by Mathias Stearn [ 19/Jul/10 ]

We can't do this as long as we want to support back to WinXP with a single binary. If you use a build with this on WinXP you get the following error before the DB starts:

"The procedure entry point SetProcessWorkingSetSizeEx could not be located in the dynamic link library KERNEL32.dll"

This happens even if you build on WinXP.

Comment by auto [ 19/Jul/10 ]

Author:

{'login': 'RedBeard0531', 'name': 'Mathias Stearn', 'email': 'mathias@10gen.com'}

Message: Revert "Try using SetProcessWorkingSetSizeEx SERVER-538"

This reverts commit f1146af710f5f5dab69927f4277ee16a72e1d0c0.
http://github.com/mongodb/mongo/commit/832601945f229d689707fdc06cde532dd051fa2f

Comment by auto [ 19/Jul/10 ]

Author:

{'login': 'RedBeard0531', 'name': 'Mathias Stearn', 'email': 'mathias@10gen.com'}

Message: Try using SetProcessWorkingSetSizeEx SERVER-538

Committing to get a win64 build.
http://github.com/mongodb/mongo/commit/f1146af710f5f5dab69927f4277ee16a72e1d0c0

Comment by AndrewK [ 04/Jun/10 ]

I should have mentioned earlier the QUOTA_LIMITS_HARDWS_MAX_ENABLE switch.

This is what I'm using in much the same way as Robert. I've been running this in production for months without any problems.

Comment by Eliot Horowitz (Inactive) [ 03/Jun/10 ]

we should try that method

Comment by Robert Wojciechowski [ 03/Jun/10 ]

I was able to limit the working set of mongod.exe to 1GB by calling SetProcessWorkingSetSizeEx on mongod.exe and setting the dwMaximumWorkingSetSize to 1GB (for example) with the flag QUOTA_LIMITS_HARDWS_MAX_ENABLE. I did this via a small c application that obtained debugger rights and set the quota based on the pid externally.

During my testing it correctly limited mongod.exe to a working set of 1GB no matter what I did in mongodb (large queries that touched many GB worth of data, etc). Before that it was eating up all available memory and causing the entire system to get sluggish.

There's no reason why this can't be made conditional depending on which version of Windows you are running. If over Vista or Server 2003 you can make a dynamic call to Kernel32.dll and set the quota.

All in all, this seems to help a lot on a Windows Server 2008 R2 box with other servers running on it.

Comment by Eliot Horowitz (Inactive) [ 10/May/10 ]

In all of our tests this had no effect.

Comment by AndrewK [ 10/May/10 ]

Any update on the last comment here? Can this be made conditional? I would think that most instances of mongo would be run on an OS newer than Vista.

Being able to to call this function via a shell command passing in both values (min/max working set) and potentially passing in the relevant flag would allow for a lot of control over how workingset is managed.

Control of this has very useful in our production environment for the past 5 months.

thanks

Comment by AndrewK [ 20/Jan/10 ]

Mathias - is there no way to make this conditional?

Also, were you able to confirm if SetProcessWorkingSetSizeEx does indeed establish a hard limit?

Comment by Mathias Stearn [ 15/Jan/10 ]

SetProcessWorkingSetSizeEx is only available in Vista and higher so we can't use it. SetProcessWorkingSetSize doesn't establish a hard limit on the working set.

Comment by Eliot Horowitz (Inactive) [ 13/Jan/10 ]

Haven't had a chance to look at yet.

Mathias, can you test and see if it works for mmap?

Comment by AndrewK [ 13/Jan/10 ]

Did SetProcessWorkingSetSizeEx() with the correct flags have more effect than SetProcessWorkingSetSize()?

Generated at Thu Feb 08 02:54:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.