[SERVER-60647] Increase of user assertions in 5.0.x Created: 13/Oct/21 Updated: 07/Feb/24 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 5.0.2, 5.0.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Tanja Großmüller | Assignee: | Backlog - Service Architecture |
| Resolution: | Unresolved | Votes: | 1 |
| Labels: | sa-remove-fv-backlog-22 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Assigned Teams: |
Service Arch
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Steps To Reproduce: | Set the log verbosity to 5 and execute the command:
In the logs you will find user assertions relating to HostUnreachable even thought the command return successfully. |
||||||||||||
| Participants: | |||||||||||||
| Case: | (copied to CRM) | ||||||||||||
| Description |
|
Since the upgrade from 4.4.4 to 5.0.3, we have observed a steadily increasing number of user assertions in the metrics. On debug level, user assertions with the error message "HostUnreachable: Connection closed by peer" became visible. As a cause for these user assertions we identified our Kubernetes readiness probe. It is successfully executed every 10 and opens a tcp socket on the mongodb port. Further analysis showed that this probe also generates error messages about "HostUnreachable" in 4.4.4, but these messages do not result in user assertions. This is the relevant log excerpt from 4.4.4:
And this is the excerpt from 5.0.3:
Please note how the marked lines appear in both versions but only in 5.0.3 there are the two additional user assertions.
As an attempt to workaround the problem we changed the tcp socket probe to the following command:
However, the user assertions persisted. Our questions: Why do both commands each return successfully but still generate a DEBUG message indicating an error?
|
| Comments |
| Comment by bovy89 N/A [ 27/Nov/23 ] | |||||||||||||||||||||||||
|
Same issue on 5.0.22 | |||||||||||||||||||||||||
| Comment by Hilariuos Punk [ 13/Oct/23 ] | |||||||||||||||||||||||||
|
Experiencing a similiar behaviour on MongoDB 6.0.5 (TLS enabled server) The below user assert error appears on a connection close via PyMongo, Go Client and Mongo shell.
This makes the monitoring of user assert errors useless.
| |||||||||||||||||||||||||
| Comment by Stefan Walluhn [ 09/Feb/22 ] | |||||||||||||||||||||||||
|
Hi, can confirm for version 5.0.6, using mongo-ruby-driver v2.16.1 as well as PyMongo 3.12.3. For Python clients I can confirm successfully query processing on client side without any assertions or connection issues.
UPDATE: Also no problems on client side for ruby clients. There isn't even a correlation between client load and user assertions within MongoDB. | |||||||||||||||||||||||||
| Comment by Tanja Großmüller [ 07/Feb/22 ] | |||||||||||||||||||||||||
|
Hi team, this issue makes the metric "db.serverStatus().asserts.user" entirely useless as it keeps rising for no reason at all. Your documentation states that this metric is supposed to show "errors that user may generate, such as out of disk space or duplicate key" and that we "can prevent these assertions by fixing a problem with your application or deployment." Is there any chance to see some progress on this in the near future? | |||||||||||||||||||||||||
| Comment by Edwin Zhou [ 25/Oct/21 ] | |||||||||||||||||||||||||
|
Thanks for your patience as we investigate this issue. I've been able to reproduce this behavior and can see clearly that there are additional user asserts in 5.0.3 compared with 4.4.10. I performed the following steps to reproduce this issue on each version:
I've attached mongod.4.4.10.log Looking at the FTDC, we see 3 uasserts in 5.0.3, up from 1 uassert in 4.4.10. I'll assign this ticket to the appropriate team for further investigation. Best, | |||||||||||||||||||||||||
| Comment by Tanja Großmüller [ 14/Oct/21 ] | |||||||||||||||||||||||||
|
@dmitry.agranat@mongodb.com Yes, the user assertions also increase when the default log verbosity of 0 is used. The number of user assertions can be retrieved from the metric "db.serverStatus().asserts.user" - so this is really the place to watch it rising (independently of the respective log message). Since the plain number does not give any insight about reason for the assertion, I increased the log verbosity level to find that out as described in your documentation https://docs.mongodb.com/manual/reference/command/serverStatus/#mongodb-serverstatus-serverstatus.asserts.
| |||||||||||||||||||||||||
| Comment by Dmitry Agranat [ 13/Oct/21 ] | |||||||||||||||||||||||||
|
grossmueller@e-spirit.com, do you see the same increase in user assertions when using the default log verbosity level? Based on the logs, you are currently using log debug levels D2, D3 and D4. |