[SERVER-29947] Implement Storage Node Watchdog Created: 30/Jun/17 Updated: 30/Oct/23 Resolved: 12/Jul/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 3.2.16, 3.4.7, 3.5.10 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Mark Benvenuto | Assignee: | Mark Benvenuto |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v3.4, v3.2
|
||||||||||||||||||||||||||||||||||||
| Sprint: | Platforms 2017-07-10, Platforms 2017-07-31 | ||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||||||
| Description |
|
Issue Status as of Jul 13, 2017 FEATURE DESCRIPTION VERSIONS OPERATION
It is an error to set watchdogPeriodSeconds at runtime if the server was not started with a value >= 60 at startup. The watchdog monitors the following directories:
If any of these directories resides in an I/O subsystem and that I/O subsystem becomes unresponsive, the watchdog will detect such condition after sufficient time has passed, then terminate mongod tearing down all its threads and exiting the process with exit code 61. The maximum time the watchdog can take to detect an unresponsive I/O subsystem is approximately twice the watchdogPeriodSeconds. IMPLEMENTATION DETAILS DIAGNOSTICS When enabled at startup, the following message will appear in the logs:
If watchdogPeriodSeconds is disabled or changed at runtime, messages like the following will appear in the logs:
At log level 1, the watchdog logs its periodic disk checks:
If the watchdog was enabled at startup, an additional section is added to the output of the serverStatus command output named "watchdog".
The meaning of this data is:
TEST METHODOLOGY Original descriptionImplement a storage node watchdog for Linux. |
| Comments |
| Comment by Githook User [ 13/Jul/17 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: |
| Comment by Githook User [ 13/Jul/17 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: |
| Comment by Githook User [ 13/Jul/17 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: (cherry picked from commit 63379590ef6fad402b17464c8ca5ad4c09a626d3) |
| Comment by Githook User [ 13/Jul/17 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: (cherry picked from commit eb333b92cae5e71affb0fe76cd388801afa8e79f) |
| Comment by Githook User [ 13/Jul/17 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: (cherry picked from commit 63379590ef6fad402b17464c8ca5ad4c09a626d3) |
| Comment by Githook User [ 13/Jul/17 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: (cherry picked from commit eb333b92cae5e71affb0fe76cd388801afa8e79f) |
| Comment by Githook User [ 12/Jul/17 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: |
| Comment by Githook User [ 12/Jul/17 ] |
|
Author: {u'username': u'markbenvenuto', u'name': u'Mark Benvenuto', u'email': u'mark.benvenuto@mongodb.com'}Message: |