[SERVER-57466] Swallow connection reset-related errors received during ASIO session establishment Created: 04/Jun/21  Updated: 29/Oct/23  Resolved: 26/Oct/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.2.0, 5.0.7

Type: Bug Priority: Major - P3
Reporter: Shane Harvey Assignee: Blake Oler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-60603 Allow connection reset errors without... Closed
Problem/Incident
causes SERVER-69063 Fix TCP keepalive option setting Closed
Related
is related to SERVER-61016 Swallow connection reset-related erro... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Sprint: Service Arch 2021-09-06, Service Arch 2021-09-20, Service Arch 2021-10-04, Service Arch 2021-11-01
Participants:
Linked BF Score: 25
Story Points: 5

 Description   

5.0.0-rc0 clusters (standalone mongod, mongos) running on macOS, I see many errors like this "ctx":"listener","msg":"Can't set socket option","attr":{"optname":"TCP_KEEPALIVE","error":"Invalid argument"}}

The log is flooded with these errors:

{"t":{"$date":"2021-06-04T13:16:52.682-07:00"},"s":"E",  "c":"NETWORK",  "id":23206,   "ctx":"listener","msg":"Can't set socket option","attr":{"optname":"TCP_KEEPALIVE","error":"Invalid argument"}}
{"t":{"$date":"2021-06-04T13:16:52.682-07:00"},"s":"W",  "c":"NETWORK",  "id":23023,   "ctx":"listener","msg":"Error accepting new connection","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"remote_endpoint: Invalid argument"}}}
{"t":{"$date":"2021-06-04T13:16:52.682-07:00"},"s":"I",  "c":"NETWORK",  "id":23018,   "ctx":"listener","msg":"Error accepting new connection on local endpoint","attr":{"localEndpoint":"127.0.0.1:27017","error":"Invalid argument"}}

The connections that error are coming from an otherwise idle load balancer (haproxy). When a driver connects directly to the mongod/s the error does not occur. What does this error mean and why do these errors occur for haproxy connections?

The load balancer is started and stopped with this run-load-balancer.sh script, like this:

$  sudo MONGODB_URI='mongodb://localhost:27017,localhost:27018/' .evergreen/run-load-balancer.sh start
Starting HAProxy...
Single Mongos LB: mongodb://127.0.0.1:8000/?loadBalanced=true
Multiple Mongos LB: mongodb://127.0.0.1:8001/?loadBalanced=true
$ sudo MONGODB_URI='mongodb://localhost:27017,localhost:27018/' .evergreen/run-load-balancer.sh stop
Stopping HAProxy...



 Comments   
Comment by Githook User [ 17/Mar/22 ]

Author:

{'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}

Message: SERVER-57466 Swallow connection reset-related errors received during ASIO session establishment

(cherry picked from commit ffededcdc670d83f4fdf5819f5ab1d9c5750c67c)

SERVER-57466 fix windows compile

(cherry picked from commit 37595d5ee16f06c964b9d416e5847b99c1a537d8)
Branch: LB
https://github.com/mongodb/mongo/commit/140b0f4016fc753c14c4e88d2decf37c37c6a60d

Comment by Githook User [ 16/Mar/22 ]

Author:

{'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}

Message: SERVER-57466 Swallow connection reset-related errors received during ASIO session establishment

(cherry picked from commit ffededcdc670d83f4fdf5819f5ab1d9c5750c67c)

SERVER-57466 fix windows compile

(cherry picked from commit 37595d5ee16f06c964b9d416e5847b99c1a537d8)
Branch: v5.0
https://github.com/mongodb/mongo/commit/ea8eccd0a2210d84493b516e470ebb3df2beca8b

Comment by Githook User [ 16/Oct/21 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-57466 fix windows compile
Branch: master
https://github.com/mongodb/mongo/commit/37595d5ee16f06c964b9d416e5847b99c1a537d8

Comment by Githook User [ 14/Oct/21 ]

Author:

{'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}

Message: SERVER-57466 Swallow connection reset-related errors received during ASIO session establishment
Branch: master
https://github.com/mongodb/mongo/commit/ffededcdc670d83f4fdf5819f5ab1d9c5750c67c

Comment by Shane Harvey [ 01/Sep/21 ]

Thanks Blake, just FYI this issue is not a blocker. The only side effect here as far as I can tell is that the server logs are flooded NETWORK errors.

Comment by Matthew Saltz (Inactive) [ 16/Jun/21 ]

Okay, thanks!

Comment by Jeffrey Yemin [ 15/Jun/21 ]

That's correct, shane.harvey

Comment by Matthew Saltz (Inactive) [ 15/Jun/21 ]

shane.harvey Would we need to fix this before 5.0 release?

Generated at Thu Feb 08 05:41:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.