[SERVER-5520] Shutdown rewrite (tracking ticket) Created: 05/Apr/12  Updated: 17/Mar/17  Resolved: 17/Mar/17

Status: Closed
Project: Core Server
Component/s: Internal Code
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Ian Whalen (Inactive) Assignee: Andy Schwerin
Resolution: Done Votes: 0
Labels: tracking-ticket
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-5773 mongo::PeriodicTask::Runner cleanup Closed
depends on CXX-103 It must be OK to run the destructors ... Closed
depends on SERVER-5703 ntservice should not have to fake out... Closed
depends on SERVER-5868 Should consolidate to a single implem... Closed
is depended on by SERVER-8707 dbclient_rs_test threading issue Closed
is depended on by TOOLS-11 mongodump exits with success despite ... Closed
is depended on by TOOLS-161 Have tools always exit with non-zero ... Closed
Related
related to SERVER-15096 Create standalone tool to read/write ... Closed
Participants:

 Description   

This ticket is the dual of SERVER-5112. The result of SERVER-5112 is that the server has a very explicit initialization system that ensures a well defined initialization order. Each subsystem can define its own startup code, and specify exactly what it depends on and what depends on it. In particular, this solved the "static initialization order" issue, in which constructors for statically defined classes run in an undefined order at before main() gets called.

The shutdown process needs similar attention. There is some "shutdown" code that needs to be run when the server is shutting down. However, this code is often run implicitly in destructors, or is lumped all together with process exit.

To start, we have duplicated definitions of the "dbexit" function, which is the entry point to the shutdown system.

One for mongos:
https://github.com/mongodb/mongo/blob/master/src/mongo/s/server.cpp#L448
And one for mongod:
https://github.com/mongodb/mongo/blob/master/src/mongo/db/instance.cpp#L1135

This function is the entry to point to the current shutdown process in the server. The fact that these signatures are identical (and used from some libraries that both mongod and mongos share) and have diverged shows that the shutdown process has not been properly factored out. "server.cpp" is the main() for mongos, and "instance.cpp" is a legacy file with old file and process related code.

There is also an "inShutdown" function, of which there are also multiple definitions:

mongod:
https://github.com/mongodb/mongo/blob/master/src/mongo/db/instance.cpp#L1135
mongos:
https://github.com/mongodb/mongo/blob/master/src/mongo/s/server.cpp#L103

This function is effectively in place as a way to notify other parts of the server that we are shutting down. However, this function is not thread safe, and it is not clear when this call is needed. This could potentially be replaced by some more explicit notification of the various components that need to be notified of shutdown.

Finally, the "dbexit" call, in addition to shutting down the database components, also exits the process. This lack of separation is a problem specifically for the tools, since some tools internally start up a database server to enable the "--dbpath" direct data file access functionality. This means that when the server gets shut down, the process exits which means the tools have limited control over how the process exits in this case. See SERVER-9473, SERVER-8064, and SERVER-1994.



 Comments   
Comment by Githook User [ 16/Oct/14 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-5520 do not check exit code when running mongod --repair in test
Branch: master
https://github.com/mongodb/mongo/commit/cacbfa2f82ee3ffe5513e727542d513d3f98ac1c

Comment by Githook User [ 29/Sep/14 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-5520 do not check exit code on tool invocation
Branch: master
https://github.com/mongodb/mongo/commit/85901e7c7601e6b969d1e553fd2ef92696683c55

Comment by Githook User [ 29/Sep/14 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-5520 SERVER-15096 do not check exit code for mongoshim in normal operation
Branch: master
https://github.com/mongodb/mongo/commit/195113534141890f15b1db931c7dfa5162dc6049

Comment by auto [ 30/Apr/12 ]

Author:

{u'login': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@10gen.com'}

Message: Do not call dbexit() from within logRotate or initLogging().

While, perhaps, not a permanent solution, using fassert() when log rotation fails is a better
choice than dbexit(), because fassert() is a fast shutdown path.

See SERVER-5520.
Branch: master
https://github.com/mongodb/mongo/commit/912bc3c25f46041ee1a69a8bef1e6b083475392d

Comment by auto [ 30/Apr/12 ]

Author:

{u'login': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@10gen.com'}

Message: Stop using mongo::dbexit() in malloc/realloc out-of-memory situations.

This is only marginally better, but is a stop gap pending a fuller startup/shutdown
rewrite.

See SERVER-5520 to track the shutdown rewrite.
Branch: master
https://github.com/mongodb/mongo/commit/a3551eadc4857ac4daf07837157909dc2d474c08

Comment by auto [ 27/Apr/12 ]

Author:

{u'login': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@10gen.com'}

Message: Remove code that has been dead since commit in June 2011.

Removes from consideration a dbexit(), so relevant to SERVER-5520.

Commit in question: 759e1e646df90b5ac858e2ccd6d587ef59912353
Branch: master
https://github.com/mongodb/mongo/commit/8dc5617a4d474d8342472f21bcb1cbebfc25b038

Comment by auto [ 25/Apr/12 ]

Author:

{u'login': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@10gen.com'}

Message: ntservice needs to call exitCleanly(), not dbexit().

SERVER-5520 – As part of a larger shutdown rewrite, exitCleanly might need
to change substantially. However, for now, exitCleanly() is the right way to
safely shut down mongod.
Branch: master
https://github.com/mongodb/mongo/commit/f460ae61beaed965a54649b703f03095a6ac055e

Comment by auto [ 25/Apr/12 ]

Author:

{u'login': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@10gen.com'}

Message: Limit exitCleanly() and ntservice module to mongos and mongod.

The notions of exitCleanly() versus dbexit() vs _exit() in the scope of mongod
shutdown need to be reconsidered. This patch just attempts to limit the scope
of the issue to mongos and mongod. It would be better to limit it to just
mongod, but the implementation of the ntservice behavior doesn't currently
support that.

SERVER-5520, SERVER-5702
Branch: master
https://github.com/mongodb/mongo/commit/127b0b78ae3ac9b2bc0a54bcd3c097c0e2a91e79

Generated at Thu Feb 08 03:09:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.