[SERVER-8586] add local/external metrics to serverStatus for network traffic Created: 15/Feb/13  Updated: 23/Feb/23  Resolved: 24/Feb/22

Status: Closed
Project: Core Server
Component/s: Diagnostics
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: Ryan Nitz Assignee: Backlog - Service Architecture
Resolution: Won't Do Votes: 0
Labels: platforms-re-triaged, re-triaged-ticket
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Service Arch
Participants:

 Description   

Currently, mongod only reports "all" network traffic on a host. This is useful, but it would also be helpful to see what the traffic is on the local and external interfaces.

"network": {
     "bytesIn": 5244015264859,
     "numRequests": 2.3714945532E10,
     "bytesOut": 12076697847847
}

E.g.

"networkLocal": {
     "bytesIn": 5244015264859,
     "numRequests": 2.3714945532E10,
     "bytesOut": 12076697847847
}

"networkExternal": {
     "bytesIn": 5244015264859,
     "numRequests": 2.3714945532E10,
     "bytesOut": 12076697847847
}

I received a system alert because a host was doing ~ 780 Mbps and I was concerned about maxing out the nic. The reality is that all of that traffic was over the local interface.



 Comments   
Comment by Lauren Lewis (Inactive) [ 24/Feb/22 ]

We haven’t heard back from you for at least one calendar year, so this issue is being closed. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Comment by Andy Schwerin [ 15/Feb/13 ]

More generally, you'd like traffic per network interface. If you had a machine with one NIC for the internet and one for the LAN, you'd probably be interested in separating those.

network: {
  bytesIn: X, // same as current
  bytesOut: Y, // same as current
  numRequests: Z, // same as current
 
  interfaces: {
    lo0: { bytesIn: Xlocal, bytesOut: Ylocal, numRequests: Zlocal },
    eth0: { ... },
    ...
  }
}

Comment by Ryan Nitz [ 15/Feb/13 ]

This format would make more sense:

"network": {
     "bytesIn": 5244015264859, // total - same number as current
     "numRequests": 2.3714945532E10, // same same
     "bytesOut": 12076697847847, // same same
 
     "bytesInLocal": 5244015264859,
     "numRequestsLocal": 2.3714945532E10,
     "bytesOutLocal": 12076697847847,
 
     "bytesInExternal": 5244015264859,
     "numRequestsExternal": 2.3714945532E10,
     "bytesOutExternal": 12076697847847
}

This data could be useful for sharding too...

Traffic over external NIC

  • A shard is doing 800 Mbps through the external interface to an app and the max on the nic is 1 Gbps
  • The balancer wants to move a chunk, but doing so would like likely cause the app to behave erratically if the nic is maxed
  • Logging an error and delaying the balance would likely be preferred to disrupting the applicaiton
  • Another possibility is throttling the rate at which data is moved

Traffic over localhost

  • A shard is doing 10 Gbps through the loopback interface and 100 Mbps through the external interface
  • The balancer wants to move a chunk, it does not matter because the localhost could probably do another 20 Gbps without disruption to the application
Comment by Ryan Nitz [ 15/Feb/13 ]

Traffic over the local interface is limited by CPU. 780 Mbps over the loopback address is trivial. As you start maxing out the external network, performance becomes inconsistant/unpredictable. If you are doing a bulk import via localhost and it spikes... it really does not matter.

Having local versus external numbers gives you finer grained control over alerting.

Comment by Eliot Horowitz (Inactive) [ 15/Feb/13 ]

Not sure it really matters?

Generated at Thu Feb 08 03:17:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.