[SERVER-6065] db.shutdownServer() does not work as expected when mongod is a Windows Service Created: 11/Jun/12  Updated: 05/Apr/17  Resolved: 10/Feb/17

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 2.0.6, 2.2.0-rc1, 3.4.0
Fix Version/s: 3.4.3, 3.5.3

Type: Bug Priority: Major - P3
Reporter: Ryan Hoffman Assignee: Mark Benvenuto
Resolution: Done Votes: 9
Labels: Windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows Server 2008 R2 Enterprise


Issue Links:
Backports
Duplicate
is duplicated by SERVER-27782 stop mongod (version 3.4.0, 3.4.1) wi... Closed
Related
is related to SERVER-6771 Windows Service: 'net stop mongodb' g... Closed
Backwards Compatibility: Fully Compatible
Operating System: Windows
Backport Requested:
v3.4
Sprint: Platforms 2017-02-13
Participants:

 Description   

Since the correct way to shutdown a service (net stop mongodb) does not work as expected in 2.0.6, I have attempted to use db.shutdownServer. This unfortionately does not work as expected. Since mongod installs itself with service failure actions to restart the service (love how it does this!), when mongo is shutdown using the shell, Windows seems to think that mongod has exited without the service being shut down, which causes it to automatically restart the service. Sample mongo shell output:

-------------------------------------
SECONDARY> db.serverStatus().uptime
531
SECONDARY> use admin
switched to db admin
SECONDARY> db.shutdownServer()
Mon Jun 11 11:07:09 DBClientCursor::init call() failed
Mon Jun 11 11:07:09 query failed : admin.$cmd

{ shutdown: 1.0 }

to: 127.0.0.1
server should be down...
Mon Jun 11 11:07:09 trying reconnect to 127.0.0.1
Mon Jun 11 11:07:10 reconnect 127.0.0.1 ok
SECONDARY> db.serverStatus().uptime
4
-------------------------------------



 Comments   
Comment by Githook User [ 28/Feb/17 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-6065 NT Service stop does not exit cleanly

(cherry picked from commit 50e9769099bb49220783f95c9045dd1259a18eb4)
Branch: v3.4
https://github.com/mongodb/mongo/commit/9bf92ae98e3621a31df91c6150f34ff3fa1213c7

Comment by Githook User [ 10/Feb/17 ]

Author:

{u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}

Message: SERVER-6065 NT Service stop does not exit cleanly
Branch: master
https://github.com/mongodb/mongo/commit/50e9769099bb49220783f95c9045dd1259a18eb4

Comment by Itzhak Kagan [ 05/Feb/17 ]

This is still occur in 3.4.2 as it reflects in 27782 (a duplicate of this bug)

Thanks,
Itzhak

Comment by Mark Benvenuto [ 20/Jan/17 ]

Verified this is still broken on 3.5.2 and 3.4.1. The service stops but incorrectly. It also fails to stop with service stop correctly.

Exit stack trace via silent process exit monitoring.

 # Child-SP          RetAddr           Call Site
00 0000009f`c9bfe5f8 00007ffa`0d7d1a10 ntdll!NtWaitForSingleObject+0x14
01 0000009f`c9bfe600 00007ffa`0d727d8f ntdll!memset+0x5090
02 0000009f`c9bff190 00007ffa`0d26ce6a ntdll!RtlExitUserProcess+0xaf
03 0000009f`c9bff1c0 00007ff6`acf1f27c kernel32!ExitProcess+0xa
04 0000009f`c9bff1f0 00007ff6`ace34b7a mongod!mongo::quickExit+0x3c [d:\mongo\src\mongo\util\quick_exit.cpp @ 110]
05 0000009f`c9bff220 00007ff6`ace348cd mongod!mongo::`anonymous namespace'::logAndQuickExit+0x8a [d:\mongo\src\mongo\util\exit.cpp @ 71]
06 0000009f`c9bff2f0 00007ff6`ab9f1805 mongod!mongo::shutdown+0x1fd [d:\mongo\src\mongo\util\exit.cpp @ 134]
07 0000009f`c9bff400 00007ff6`ace8b6f0 mongod!mongo::exitCleanly+0x25 [d:\mongo\src\mongo\util\exit.h @ 82]
08 0000009f`c9bff430 00007ff6`ace89130 mongod!<lambda_47b7b45126974acccccd84fb5443422b>::operator()+0x60 [d:\mongo\src\mongo\util\ntservice.cpp @ 539]
09 0000009f`c9bff4a0 00007ff6`ace8a2d0 mongod!std::_Invoker_functor::_Call<<lambda_47b7b45126974acccccd84fb5443422b> & __ptr64>+0x30 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1377]
0a 0000009f`c9bff4d0 00007ff6`ace894a4 mongod!std::invoke<<lambda_47b7b45126974acccccd84fb5443422b> & __ptr64>+0x30 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1445]
0b 0000009f`c9bff500 00007ff6`ace8c555 mongod!std::_Invoke_ret<bool,<lambda_47b7b45126974acccccd84fb5443422b> & __ptr64>+0x34 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1469]
0c 0000009f`c9bff530 00007ff6`ace8b7d4 mongod!std::_Func_impl<<lambda_47b7b45126974acccccd84fb5443422b>,std::allocator<int>,bool>::_Do_call+0x35 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 214]
0d 0000009f`c9bff570 00007ff6`ace8c09a mongod!std::_Func_class<bool>::operator()+0x54 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 280]
0e 0000009f`c9bff5b0 00007ff6`ace8b8e5 mongod!std::_Packaged_state<bool __cdecl(void)>::_Call_immediate+0x3a [c:\program files (x86)\microsoft visual studio 14.0\vc\include\future @ 609]
0f 0000009f`c9bff620 00007ff6`ace89180 mongod!std::packaged_task<bool __cdecl(void)>::operator()+0x75 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\future @ 1823]
10 0000009f`c9bff670 00007ff6`ace8a320 mongod!std::_Invoker_functor::_Call<std::packaged_task<bool __cdecl(void)> >+0x30 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1377]
11 0000009f`c9bff6a0 00007ff6`ace892ac mongod!std::invoke<std::packaged_task<bool __cdecl(void)> >+0x30 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1445]
12 0000009f`c9bff6d0 00007ff6`ace8d23a mongod!std::_LaunchPad<std::unique_ptr<std::tuple<std::packaged_task<bool __cdecl(void)> >,std::default_delete<std::tuple<std::packaged_task<bool __cdecl(void)> > > > >::_Execute<0>+0x3c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 241]
13 0000009f`c9bff700 00007ff6`ace8cba8 mongod!std::_LaunchPad<std::unique_ptr<std::tuple<std::packaged_task<bool __cdecl(void)> >,std::default_delete<std::tuple<std::packaged_task<bool __cdecl(void)> > > > >::_Run+0x6a [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247]
14 0000009f`c9bff750 00007ff6`ab77539d mongod!std::_LaunchPad<std::unique_ptr<std::tuple<std::packaged_task<bool __cdecl(void)> >,std::default_delete<std::tuple<std::packaged_task<bool __cdecl(void)> > > > >::_Go+0x28 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 233]
15 0000009f`c9bff780 00007ff9`b9bda1bc mongod!std::_Pad::_Call_func+0x2d [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210]
16 0000009f`c9bff7b0 00007ff9`b9bd9e13 ucrtbased!register_onexit_function+0x47c
17 0000009f`c9bff7f0 00007ffa`0d258364 ucrtbased!register_onexit_function+0xd3
18 0000009f`c9bff840 00007ffa`0d7870d1 kernel32!BaseThreadInitThunk+0x14
19 0000009f`c9bff870 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Comment by Tad Marshall [ 20/Aug/12 ]

This bug will be fixed when SERVER-6771 is fixed. I'm keeping the bugs separate because the symptoms and steps-to-reproduce are different, but the fix for both bugs is to unwind (return) from our shutdown code and not exit directly from it. The caller will still call _exit(), so we preserve that desired change to our exit logic.

Comment by Tad Marshall [ 20/Aug/12 ]

The issue with "net stop mongodb" is just an unwanted error message ("System error 109 has occurred." and "The pipe has been ended.") and isn't a symptom of anything worse going on. See https://jira.mongodb.org/browse/SERVER-6771 for some information about what is happening.

This bug is sort of the flipside of the other one. In SERVER-6771, the Windows Service Control Manager tells us to shut down, and we do that and indicate that we are stopped, but we then exit before returning to the SCM the way it expects us to. In this bug, we exit before we ever reach the code that would tell the SCM that we are shutting down, so it doesn't know that we have exited on purpose and so it restarts us (if set to do so on "failure").

Both bugs need basically the same fix. When shutting down while running as a service, we need to return from the "shutdown" code and not exit from it. Once we have unwound through the SCM, including telling it that we have shut down (therefore that we have not "failed"), we are then free to exit the process.

Despite the error message that you get from "net stop mongodb", that is probably the better way to shut down MongoDB when it is running as a service (until this ticket is fixed), because we at least tell the Windows Service Control Manager that we are stopped before we exit. With "net stop mongodb" you won't get an error logged to the Windows Event Log and you won't trigger a restart if the service is set to restart on failure.

But either way of shutting down is safe and is a clean shutdown from MongoDB's point of view. Journal files will be closed and cleaned up and the mongod.lock files will be truncated, so except for annoying the Windows Service Control Manager everything is fine.

Both of these bugs were created when we changed our exit logic to exit immediately after shutting down instead of unwinding through all calling routines and exiting from main() (the way we used to).

Generated at Thu Feb 08 03:10:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.