[SERVER-17140] ReplExecNetThread, errno:99 Cannot assign requested address Created: 31/Jan/15  Updated: 26/Feb/15  Resolved: 19/Feb/15

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.8.0-rc3, 3.0.0-rc7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: rgpublic Assignee: Scott Hernandez (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File bug_diesel2.log.gz     File bug_diesel_clean.log.gz    
Issue Links:
Related
is related to SERVER-10710 No new connections accepted under hea... Closed
Backwards Compatibility: Fully Compatible
Sprint: RPL 0 3/13/15
Participants:

 Description   

I'm currently running 2.8.0-RC3. Now, I've set up a second server with RC7. I've read that the on-disk index format is different between these two version and I need to start with --repair. But I assume that this is not necessary (yet), because RC7 is empty and doesnt have any indexes yet. I thought the use of a replica set would be a good way to upgrade without service interruption. So, now I connect to RC3 (Primary) and say rs.add("otherserver"). At first this seemed to work and the other server is in "STARTUP2" mode as usual when it's starting sync. But on the primary (as I later found out) the log filled with "ReplExecNetThread-0, Failed to connect to x.x.x.x, errno:99 Cannot assign requested address" errors. After some minutes the primary suddenlty becomes SECONDARY and the database becomes unusable, because there is no PRIMARY I needed some time to find out you need to remove the bad server from the replica set config and do a reconfigure with

{force:true}

to recover from that situation. Is this a bug or am I doing something wrong? I've setup replica sets earlier with previous Mongo version and it worked flawlessly everytime. The ports are reachable in either direction (tested with telnet x.x.x.x 27017).



 Comments   
Comment by Jeremy Mikola [ 23/Feb/15 ]

rgpublic: If the "replicaSet" option seems to cause a rise in spurious connections, my first thought would be that the hostname in your connection string (i.e. MongoClient's constructor) doesn't match the hostname in the replica set configuration, which the driver sees when issuing an isMaster command. While the driver does connect to the host(s) in the URI, it only persists connections where the host/port match the replica set configuration. If you'd like to debug this further, I'd be happy to help but would suggest you open a new ticket in the PHP project and include as much information from MongoLog (the driver's internal logging mechanism) as you can.

Comment by rgpublic [ 23/Feb/15 ]

Jeremy, thank you looking into this issue. I've removed the connect=>true, thanks for the hint. Don't know where I got that from - it's in our code for ages. I've added the ->close(true) only recently because I discovered it somewhere on the web in a deperate attempt to solve the problem. I've noticed that it's discouraged, but it was worth trying for me. Nevertheless, the problem appears with or without the close().

So, basically the problem is as follows: We got a web server with lots of different PHP scripts running every second, every minute. They run only a very short period of time (only a second or even less). All of these scripts include the very same file "database.php" which creates the connection mentioned above. We've got a replica set with a primary (on which these scripts are running) and a secondary. Everything syncs and works beautifully. As soon as I add "replicaSet"=>"diesel" to the connection string the TIME_WAIT connections rise like crazy. After a minute there are sometimes already 10000 connections or even more.
If I am quick enough to remove the replicaSet option again the TIME_WAIT number begins to fall. If I wait a little bit longer though, the connections max out. In the mongo log the primary says it can't replicate properly. After that it all falls apart very quickly and the database is offline

Comment by Jeremy Mikola [ 23/Feb/15 ]

Investigating and reproducing with Python

After some investigation, I believe this issue is simply due to a high connection load. I attempted to reproduce the TIME_WAIT build-up using a driver other than PHP and came up with the following simple Python script:

from pymongo import MongoClient
from time import sleep
 
conns = []
 
for x in range(0, 10):
	conns.append(MongoClient('localhost', 10001));
 
sleep(1)
 
for c in conns:
	c.disconnect()

In the script, I'm just opening a bunch of connections and then closing them all after a second. The test was run twice (against 2.6.7 and 3.0.0-rc9) using two-member replica sets with a primary on localhost:10001 and secondary on localhost:10002. I observed the same results with both server versions.

In a separate terminal, I was running watch -n 0.5 'netstat -nap|grep 10001' to keep tabs on the sockets of interest. Before running the test, I saw a few connections between the replica set members:

tcp        0      0 0.0.0.0:10002           0.0.0.0:*               LISTEN      26356/mongod
tcp        0      0 127.0.0.1:36297         127.0.1.1:10002         ESTABLISHED 26292/mongod
tcp        0      0 127.0.0.1:33054         127.0.1.1:10002         ESTABLISHED 26292/mongod
tcp        0      0 127.0.0.1:33053         127.0.1.1:10002         ESTABLISHED 26292/mongod
tcp        0      0 127.0.0.1:36226         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.1.1:10002         127.0.0.1:36297         ESTABLISHED 26356/mongod
tcp        0      0 127.0.0.1:36183         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.1.1:10002         127.0.0.1:33054         ESTABLISHED 26356/mongod
tcp        0      0 127.0.1.1:10002         127.0.0.1:33053         ESTABLISHED 26356/mongod

Running the script once would create an influx of established connections between the python process and our primary server, and those connections would quickly close and end up in a TIME_WAIT state:

tcp        0      0 0.0.0.0:10002           0.0.0.0:*               LISTEN      26356/mongod
tcp        0      0 127.0.0.1:36356         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:36360         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:36357         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:36339         127.0.1.1:10002         ESTABLISHED 26292/mongod
tcp        0      0 127.0.0.1:36363         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:36364         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:36297         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:33054         127.0.1.1:10002         ESTABLISHED 26292/mongod
tcp        0      0 127.0.0.1:33053         127.0.1.1:10002         ESTABLISHED 26292/mongod
tcp        0      0 127.0.0.1:36226         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:36362         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:36361         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:36355         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:36358         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.0.1:36359         127.0.1.1:10002         TIME_WAIT   -
tcp        0      0 127.0.1.1:10002         127.0.0.1:36339         ESTABLISHED 26356/mongod
tcp        0      0 127.0.1.1:10002         127.0.0.1:33054         ESTABLISHED 26356/mongod
tcp        0      0 127.0.1.1:10002         127.0.0.1:33053         ESTABLISHED 26356/mongod

Running the script multiple times in succession would create ten new connections, which would then end up in TIME_WAIT. The effect was purely cumulative, so I'll spare you another paste.

Further reading

This prompted me to refresh my knowledge of TCP states. I found some pertinent discussion about accumulating TIME_WAIT sockets in this Server Fault thread, and this answer mentioned that it's a symptom of many short-lived connections. Additionally, Coping with the TCP TIME-WAIT state on busy Linux servers provided a great overview of TIME_WAIT, ways to get around it, and reasons why you probably shouldn't.

Back to your PHP script

Reviewing the thread, the first thing that stood out to me was this code example you shared:

$mongo=new MongoClient("mongodb://diesel.kryd.int", ["connect"=>true, "replicaSet" => "diesel"]);
$mongo->close(true);

Firstly, the "connect" option defaults to true, so there's no need to specify it when constructing MongoClient.

Secondly, and more importantly, there should be no reason to call MongoClient::close() in your application (we even say that in the documentation). Forcefully closing the connections circumvents the driver's ability to use persistent connections. This doesn't jive well with a high-traffic web application where each incoming request needs to open a connection to MongoDB. Assuming the operating system leaves closed sockets in TIME_WAIT for 60 seconds (the default I observed), I would expect you to hit the limit you've cited for any non-trivial request load.

Please remove the close() call and let us know if that resolves the resource exhaustion.

Additionally, as a developer on the PHP driver, I'd be curious to know what lead you to start using close(). The documentation has discouraged its use since persistent connections (and connection pooling before then) was added in the 1.2.x era. That said, we still find people using it from time to time. Ultimately, we should remove it in the next major version of the driver (when we can break BC for the public API).

Comment by rgpublic [ 22/Feb/15 ]

PS: Now that my secondary finally is in state "SECONDARY" I tried the above two-line script again and, astonishingly this still leads to the CLOSE_WAIT counter incrementing... What am I doing wrong here? I'm puzzled. I've updated both server to 3.0.0-rc9 in the meantime BTW - to no avail. The version of my PHP driver is "1.6.3".

Comment by rgpublic [ 22/Feb/15 ]

I think I've finally found the bug. The problem is: If the other server (secondary) is still in STARTUP2 phase and you connect from PHP with the replicaSet option set then the connection will remain in CLOSE_WAIT state. I use this PHP script:

$mongo=new MongoClient("mongodb://diesel.kryd.int",Array("connect"=>true,"replicaSet"=>"diesel"));
$mongo->close(true);

The close wait counter ("netstat | grep diesel2 | grep WAIT | wc -l") will go up 1 with each call of this script. If you have an active webserver with lots of running scripts, the counter quickly goes up and is using all ephemeral ports. Afterwards, the replication starts to fail.

Comment by Scott Hernandez (Inactive) [ 12/Feb/15 ]

I believe that what might be happening is that the php client (driver) is making the extra connections, not the mongodb server. It seems like from the connection logging, before you add the other node to the replica set, that there are a lot of connection being created and used from outside of the replica set. This is most likely also the process which is not correctly closing connections, leaving resources unreleased until the operating system can clean them up. This is ultimately what causes the underlying issue of running out of file handles and assignable addresses.

I think you will want to look into your php environment and process management in order to figure out what is going on. Depending on your mongodb php version you may just be able to upgrade if this is a known issue, but that would be a question for the php experts and maintainers. I expect if you look in your php server logs you may very well see errors as well, and they may help to understand better what is going on.

In short, I can see nothing in server logs showing the replica set servers as being the cause of the high number of connections, nor are there any signs that any replica set connections have errors until the final problem. This all points to the issue being outside of the mongodb servers, and their errors being symptoms of another problem, and not the cause.

Comment by rgpublic [ 06/Feb/15 ]

About your further questions:

1) Yes, I'm running PHP on the same server. This is how I connect:

<?
$options=Array();
$options["replicaSet"]="diesel";
$options["connect"]=true;
$options["w"]=0;
$options["socketTimeoutMS"]=-1;
 
$conn=new MongoClient("mongodb://diesel.kryd.int",$options);
?>

2) And, yes again, I've got a single node before running rs.add();

Comment by rgpublic [ 06/Feb/15 ]

Here are the corresponding files from both servers with all queries removed

Comment by Scott Hernandez (Inactive) [ 03/Feb/15 ]

Are you running php on the same server as the database server, diesel? What is your connection string being used in your application?

If you want to remove the QUERY lines that would be fine for the mongodb server log files. The logs should compress nicely so anything under 100MB per file compressed should be fine.

So, before you run rs.add(...), what replica set config do you have, a single node?

Comment by rgpublic [ 03/Feb/15 ]

Between the before and after output I logged in to the mongo shell on the primary and did a rs.add("diesel2.kryd.int"). That's it.
Yes, the database on the primary is under active use by various PHP client applications connected to the database and doing queries. Other major applications running on the same server are apache/php and elasticsearch.
Could you please explain what you mean by "provide logs from both hosts during the time that the connections grow". You mean the Mongo logs? The logs contain many queries etc. Do you need only the REPL part or anything else I might grep to keep log size under control and omit possibly confidential data?

Comment by Scott Hernandez (Inactive) [ 03/Feb/15 ]

This does show that os is waiting for those sockets to timeout so they can be cleaned up.

Also, please answer the outstanding questions:

Can you provide logs from both hosts during the time that the connections grow?

What did you do between the before and after output?

Do you have any other clients/applications running on diesel (148.251.187.175)?

Comment by rgpublic [ 03/Feb/15 ]

I tried the mongo client and I can connect to the other server:

root@diesel ~ # mongo --host 148.251.187.176:27017
MongoDB shell version: 2.8.0-rc3
connecting to: 148.251.187.176:27017/test
Server has startup warnings: 
2015-02-03T18:58:38.966+0100 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
...

With netstat -nap the primary again shows too many lines to enter here, they look like below. What I find astonishing, I just found out that they now seem to be always there - even before I add the secondary (perhaps from the last run)? And even after I enter rs.remove("diesel2.kryd.int:27017") they never get removed. I'm no expert though - perhaps this has already been obvious to you all along...

tcp        0      0 148.251.187.175:57997   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:43957   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:40488   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:47507   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:40714   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:56682   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:41715   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:60214   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:53598   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:36778   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:41292   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:41069   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:41213   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:36811   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:59604   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:48919   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:44659   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:37628   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:34471   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:48015   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:48098   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:56511   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:33489   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:43657   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:50702   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:60260   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:44349   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:48443   148.251.187.175:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:41170   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:57769   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:42020   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:39919   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:44718   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:42189   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:59448   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:53249   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:42818   148.251.187.176:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.175:52258   148.251.187.176:27017   TIME_WAIT   -    

The full result of the secondary is:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1552/sshd       
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN      26910/mongod    
tcp        0     36 148.251.187.176:22      78.46.174.247:40065     ESTABLISHED 26931/4         
tcp        0      0 148.251.187.176:41931   148.251.187.175:27017   TIME_WAIT   -               
tcp        0      0 127.0.0.1:38181         127.0.0.1:27017         ESTABLISHED 26926/mongo     
tcp        0      0 148.251.187.176:27017   148.251.187.175:58013   ESTABLISHED 26910/mongod    
tcp        0      0 148.251.187.176:41935   148.251.187.175:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.176:41933   148.251.187.175:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.176:22      78.46.174.247:60821     ESTABLISHED 8712/2          
tcp        0      0 148.251.187.176:27017   148.251.187.175:35774   ESTABLISHED 26910/mongod    
tcp        0      0 148.251.187.176:22      78.46.174.247:60804     ESTABLISHED 24289/0         
tcp        0      0 148.251.187.176:27017   148.251.187.175:35953   ESTABLISHED 26910/mongod    
tcp        0      0 148.251.187.176:22      78.46.174.247:40043     ESTABLISHED 26792/3         
tcp        0      0 127.0.0.1:27017         127.0.0.1:38181         ESTABLISHED 26910/mongod    
tcp        0      0 148.251.187.176:41932   148.251.187.175:27017   TIME_WAIT   -               
tcp        0      0 148.251.187.176:41937   148.251.187.175:27017   ESTABLISHED 26910/mongod    
tcp        0      0 148.251.187.176:41934   148.251.187.175:27017   ESTABLISHED 26910/mongod    
tcp        0      0 148.251.187.176:41936   148.251.187.175:27017   ESTABLISHED 26910/mongod    
tcp6       0      0 :::22                   :::*                    LISTEN      1552/sshd       
udp        0      0 148.251.187.176:123     0.0.0.0:*                           1255/ntpd       
udp        0      0 127.0.0.1:123           0.0.0.0:*                           1255/ntpd       
udp        0      0 0.0.0.0:123             0.0.0.0:*                           1255/ntpd       
udp6       0      0 2a01:4f8:211:928::2:123 :::*                                1255/ntpd       
udp6       0      0 fe80::ec4:7aff:fe07:123 :::*                                1255/ntpd       
udp6       0      0 ::1:123                 :::*                                1255/ntpd       
udp6       0      0 :::123                  :::*                                1255/ntpd       
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     823198   26910/mongod        /tmp/mongodb-27017.sock
unix  2      [ ACC ]     STREAM     LISTENING     7778     637/dbus-daemon     /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     7534     1/init              @/com/ubuntu/upstart
unix  2      [ ACC ]     SEQPACKET  LISTENING     7576     423/systemd-udevd   /run/udev/control
unix  10     [ ]         DGRAM                    15530    597/rsyslogd        /dev/log
unix  2      [ ACC ]     STREAM     LISTENING     11465    977/acpid           /var/run/acpid.socket
unix  3      [ ]         STREAM     CONNECTED     7981     1/init              @/com/ubuntu/upstart
unix  3      [ ]         STREAM     CONNECTED     15546    637/dbus-daemon     
unix  3      [ ]         STREAM     CONNECTED     7758     1/init              @/com/ubuntu/upstart
unix  3      [ ]         DGRAM                    15490    423/systemd-udevd   
unix  2      [ ]         DGRAM                    495828   24289/0             
unix  2      [ ]         DGRAM                    15545    637/dbus-daemon     
unix  2      [ ]         DGRAM                    7900     977/acpid           
unix  2      [ ]         DGRAM                    543669   8712/2              
unix  2      [ ]         DGRAM                    823587   26931/4             
unix  3      [ ]         STREAM     CONNECTED     13359    413/upstart-udev-br 
unix  3      [ ]         STREAM     CONNECTED     11438    637/dbus-daemon     /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     12511    1/init              @/com/ubuntu/upstart
unix  3      [ ]         STREAM     CONNECTED     15548    637/dbus-daemon     /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    7990     1255/ntpd           
unix  3      [ ]         DGRAM                    15489    423/systemd-udevd   
unix  3      [ ]         STREAM     CONNECTED     15547    637/dbus-daemon     
unix  3      [ ]         STREAM     CONNECTED     9458     1245/upstart-socket 
unix  2      [ ]         DGRAM                    823523   26792/3             
unix  3      [ ]         STREAM     CONNECTED     8299     1/init              
unix  2      [ ]         DGRAM                    7808     726/systemd-logind  
unix  3      [ ]         STREAM     CONNECTED     1532     573/upstart-file-br 
unix  3      [ ]         STREAM     CONNECTED     7827     726/systemd-logind  

Comment by Scott Hernandez (Inactive) [ 02/Feb/15 ]

Yep, seems to be something creating those connections, and then they don't get closed quickly enough, leading to resources being limited/spent.

If your run netstat with the additional "-p" flag it will show the program name, and other stats.

Can you provide logs from both hosts during the time that the connections grow?
What did you do between the before and after output?
Can you use the mongo client, instead of telnet, to make the connection to validate that it works (mongo --host 148.251.187.176:27017)?
Do you have any other clients/applications running on diesel (148.251.187.175)?

Comment by rgpublic [ 02/Feb/15 ]

OK, here we go:

BEFORE
 
root@diesel /opt/mongo/bin # netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:21100           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:30000           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:6002            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:5267            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:30100           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:16500           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:31000           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:16600           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:31100           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:17500           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:17600           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:20000           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:20100           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:21000           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:51821         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35302   ESTABLISHED
tcp        0      0 127.0.0.1:52000         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35233   54.228.246.64:21000     TIME_WAIT  
tcp        0      0 127.0.0.1:31000         127.0.0.1:50191         ESTABLISHED
tcp        0      0 127.0.0.1:52297         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52199         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:34928   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59610   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59611   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:21100         127.0.0.1:38331         ESTABLISHED
tcp        0      0 127.0.0.1:52056         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51874         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52284         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59468   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51926         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35483   ESTABLISHED
tcp        0      0 127.0.0.1:51644         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52120         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59363   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35376   ESTABLISHED
tcp        0      0 148.251.187.175:59243   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59606   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59384   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51792         127.0.0.1:31000         TIME_WAIT  
tcp        0    251 148.251.187.175:59224   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52252         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59365   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52330         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:50621         127.0.0.1:21000         TIME_WAIT  
tcp        0      0 127.0.0.1:52295         127.0.0.1:31000         TIME_WAIT  
tcp        0    311 148.251.187.175:51809   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59486   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51806         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:44930   ESTABLISHED
tcp        0      0 127.0.0.1:52209         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:48210   ESTABLISHED
tcp        0      0 127.0.0.1:31000         127.0.0.1:44474         ESTABLISHED
tcp        0      0 127.0.0.1:52174         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:50346         127.0.0.1:21000         TIME_WAIT  
tcp        0      0 127.0.0.1:50212         127.0.0.1:16500         TIME_WAIT  
tcp        0      0 127.0.0.1:51665         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51778         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:48167         127.0.0.1:31000         ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:35315   ESTABLISHED
tcp        0      0 148.251.187.175:35458   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59353   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:48162   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:34999   54.228.246.64:21000     TIME_WAIT  
tcp        0      0 127.0.0.1:52287         127.0.0.1:31000         TIME_WAIT  
tcp        0     60 148.251.187.175:27017   148.251.187.175:58985   ESTABLISHED
tcp        0      0 127.0.0.1:36838         127.0.0.1:20000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:43643   ESTABLISHED
tcp        0      0 127.0.0.1:31000         127.0.0.1:52239         ESTABLISHED
tcp        0      0 148.251.187.175:59131   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:60844   ESTABLISHED
tcp        0      0 127.0.0.1:51929         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51930         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35531   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52194         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51965         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:50459         127.0.0.1:21000         TIME_WAIT  
tcp        0      0 148.251.187.175:54373   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:35477   54.228.246.64:21000     TIME_WAIT  
tcp        0      0 127.0.0.1:52109         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:16600         127.0.0.1:33593         ESTABLISHED
tcp        0      0 127.0.0.1:50731         127.0.0.1:21000         TIME_WAIT  
tcp        0      0 148.251.187.175:43180   148.251.187.175:80      TIME_WAIT  
tcp        0      0 148.251.187.175:59470   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52134         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59496   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:57116   148.251.187.175:27017   ESTABLISHED
tcp        0     49 148.251.187.175:27017   148.251.187.175:44661   ESTABLISHED
tcp        0      0 127.0.0.1:52137         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51777         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59112   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52280         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52065         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35315   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:51920         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59506   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35408   ESTABLISHED
tcp        0      0 127.0.0.1:52204         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:36055   ESTABLISHED
tcp        0      0 127.0.0.1:52187         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:49055         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52037         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59347   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:43643   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:51809         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:57494         127.0.0.1:30000         TIME_WAIT  
tcp        0      0 148.251.187.175:59495   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35356   ESTABLISHED
tcp        0      0 148.251.187.175:59614   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59128   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:41905   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:57794   ESTABLISHED
tcp        0      0 148.251.187.175:35416   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:35423   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:58982   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:49099   ESTABLISHED
tcp        0      0 148.251.187.175:35450   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:35521   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:35548   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:35430   ESTABLISHED
tcp        0      0 148.251.187.175:59500   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51939         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:36061   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:36053   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:60674   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:57484   ESTABLISHED
tcp        0      0 148.251.187.175:59488   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35531   ESTABLISHED
tcp        0      0 127.0.0.1:52261         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59585   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:43892         127.0.0.1:31100         ESTABLISHED
tcp        0      0 148.251.187.175:35574   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52318         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59154   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:37143   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52031         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59111   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51774         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:53571         127.0.0.1:17500         TIME_WAIT  
tcp        0      0 148.251.187.175:59361   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35544   ESTABLISHED
tcp        0      0 127.0.0.1:52008         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:31000         127.0.0.1:37326         ESTABLISHED
tcp        0      0 127.0.0.1:51853         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:53677   ESTABLISHED
tcp        0      0 148.251.187.175:47186   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:33593         127.0.0.1:16600         ESTABLISHED
tcp        0      0 127.0.0.1:52022         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52229         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59151   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:34936   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:31000         127.0.0.1:39120         ESTABLISHED
tcp        0      0 148.251.187.175:53604   148.251.187.175:4949    TIME_WAIT  
tcp        0      0 127.0.0.1:52049         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59121   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51793         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52272         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51802         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51817         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59249   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:38880   ESTABLISHED
tcp        0      0 148.251.187.175:59385   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52141         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35376   148.251.187.175:27017   ESTABLISHED
tcp        0     36 148.251.187.175:22      78.46.174.247:55944     ESTABLISHED
tcp        0      0 127.0.0.1:51925         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59603   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51938         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59234   148.251.187.175:27017   TIME_WAIT  
tcp        0    210 148.251.187.175:44661   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:35483   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52014         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59478   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51830         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:36063   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:36053   ESTABLISHED
tcp        0      0 148.251.187.175:34927   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:51940         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51935         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59239   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:31000         127.0.0.1:48167         ESTABLISHED
tcp        0      0 148.251.187.175:59237   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59116   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51784         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52292         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59125   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:55459   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:51923         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51969         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35985   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:50396   ESTABLISHED
tcp        0      0 148.251.187.175:59594   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52290         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:36063   ESTABLISHED
tcp        0      0 148.251.187.175:59469   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51879         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35304   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52055         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35399   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52305         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:43327   148.251.187.175:80      TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35736   ESTABLISHED
tcp        0      0 127.0.0.1:50843         127.0.0.1:21000         TIME_WAIT  
tcp        0      0 148.251.187.175:35319   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:51812         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52278         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:57116   ESTABLISHED
tcp        0      0 148.251.187.175:59250   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52170         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:49099   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59578   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59391   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35511   ESTABLISHED
tcp        0      0 127.0.0.1:52286         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:56388   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:36892   ESTABLISHED
tcp        0      0 127.0.0.1:57598         127.0.0.1:30000         TIME_WAIT  
tcp        0      0 148.251.187.175:59226   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51808         127.0.0.1:31100         ESTABLISHED
tcp        0      0 127.0.0.1:51869         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59493   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:43286   148.251.187.175:80      TIME_WAIT  
tcp        0      0 127.0.0.1:50579         127.0.0.1:21000         TIME_WAIT  
tcp        0      0 148.251.187.175:35518   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59512   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:31000         127.0.0.1:51650         ESTABLISHED
tcp        0      0 127.0.0.1:51906         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:50396   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:35307   ESTABLISHED
tcp        0      0 148.251.187.175:59476   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52046         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51947         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35736   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52175         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59586   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:35430   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:40900   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59147   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:51809   ESTABLISHED
tcp        0      0 127.0.0.1:31000         127.0.0.1:51825         TIME_WAIT  
tcp        0      0 148.251.187.175:45694   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:35518   ESTABLISHED
tcp        0      0 127.0.0.1:51864         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:43052   148.251.187.175:80      TIME_WAIT  
tcp        0      0 127.0.0.1:52249         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35458   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:40900   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:37245   ESTABLISHED
tcp        0      0 127.0.0.1:52190         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52015         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52239         127.0.0.1:31000         ESTABLISHED
tcp        0      0 127.0.0.1:52010         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52093         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52219         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52040         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52034         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52057         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51823         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:55459   ESTABLISHED
tcp        0      0 148.251.187.175:43053   148.251.187.175:80      TIME_WAIT  
tcp        0      0 148.251.187.175:59159   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:53459         127.0.0.1:17500         TIME_WAIT  
tcp        0      0 148.251.187.175:59370   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59636   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59617   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52016         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:31100         127.0.0.1:51808         ESTABLISHED
tcp        0      0 148.251.187.175:59236   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51650         127.0.0.1:31000         ESTABLISHED
tcp        0      0 127.0.0.1:51984         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:43285   148.251.187.175:80      TIME_WAIT  
tcp        0      0 148.251.187.175:59138   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:35613   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59160   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52043         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52145         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59629   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:52488   ESTABLISHED
tcp        0      0 127.0.0.1:51931         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:41905   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:46448   ESTABLISHED
tcp        0      0 127.0.0.1:31000         127.0.0.1:51800         TIME_WAIT  
tcp        0      0 148.251.187.175:46380   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:44474         127.0.0.1:31000         ESTABLISHED
tcp        0      0 127.0.0.1:31100         127.0.0.1:32825         ESTABLISHED
tcp        0      0 127.0.0.1:52299         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:48210   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:35486   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:36892   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:51917         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52323         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52298         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59625   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59518   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:37214         127.0.0.1:20000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:37143   ESTABLISHED
tcp        0      0 148.251.187.175:37245   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:51994         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52142         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35436   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52193         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51919         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:34886   54.228.246.64:20000     TIME_WAIT  
tcp        0      0 127.0.0.1:53083         127.0.0.1:17500         TIME_WAIT  
tcp        0      0 148.251.187.175:59521   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59595   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35218   ESTABLISHED
tcp        0      0 148.251.187.175:35226   54.228.246.64:20000     TIME_WAIT  
tcp        0      0 127.0.0.1:57706         127.0.0.1:30000         TIME_WAIT  
tcp        0      0 148.251.187.175:59230   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:39399   ESTABLISHED
tcp        0      0 127.0.0.1:52165         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59110   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:58985   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:35110   54.228.246.64:20000     TIME_WAIT  
tcp        0      0 127.0.0.1:51893         127.0.0.1:31000         ESTABLISHED
tcp        0      0 148.251.187.175:59357   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:36055   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:31100         127.0.0.1:43892         ESTABLISHED
tcp        0    223 148.251.187.175:46448   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:60674   ESTABLISHED
tcp        0      0 148.251.187.175:59513   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:35511   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:31000         127.0.0.1:52240         ESTABLISHED
tcp        0      0 127.0.0.1:52059         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:34936   ESTABLISHED
tcp        0      0 127.0.0.1:31000         127.0.0.1:51978         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35423   ESTABLISHED
tcp       25      0 127.0.0.1:31000         127.0.0.1:46831         CLOSE_WAIT 
tcp        0      0 127.0.0.1:36942         127.0.0.1:20000         TIME_WAIT  
tcp        0      0 127.0.0.1:50191         127.0.0.1:31000         ESTABLISHED
tcp        0      0 148.251.187.175:22      78.46.174.247:34211     ESTABLISHED
tcp        0      0 148.251.187.175:35146   54.228.246.64:21000     TIME_WAIT  
tcp        0      0 148.251.187.175:59228   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52157         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52237         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59501   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:36071   ESTABLISHED
tcp        0      0 148.251.187.175:59153   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:45694   ESTABLISHED
tcp        0      0 127.0.0.1:52267         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:53677   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52341         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52104         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:36057   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52240         127.0.0.1:31000         ESTABLISHED
tcp        0      0 148.251.187.175:59484   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52300         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52214         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:37050         127.0.0.1:20000         TIME_WAIT  
tcp        0      0 127.0.0.1:51643         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52262         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52131         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52168         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59473   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51775         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:31000         127.0.0.1:52251         TIME_WAIT  
tcp        0      0 127.0.0.1:52154         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59134   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:48162   ESTABLISHED
tcp        0      0 127.0.0.1:51959         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:59224   ESTABLISHED
tcp        0      0 127.0.0.1:51824         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35548   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:36061   ESTABLISHED
tcp        0      0 127.0.0.1:42905         127.0.0.1:30100         ESTABLISHED
tcp        0      0 127.0.0.1:52050         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52158         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51797         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:31000         127.0.0.1:59780         ESTABLISHED
tcp        0      0 148.251.187.175:60844   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:59578   ESTABLISHED
tcp        0      0 127.0.0.1:51847         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51840         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52115         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59375   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52159         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51818         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52039         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35574   ESTABLISHED
tcp        0      0 127.0.0.1:38331         127.0.0.1:21100         ESTABLISHED
tcp        0      0 127.0.0.1:52224         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35985   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:56128   ESTABLISHED
tcp        0      0 148.251.187.175:35218   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:35307   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52152         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59373   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52310         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:34987   54.228.246.64:20000     TIME_WAIT  
tcp        0      0 127.0.0.1:50567         127.0.0.1:21000         TIME_WAIT  
tcp        0      0 148.251.187.175:57794   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:31100         127.0.0.1:49357         ESTABLISHED
tcp        0      0 148.251.187.175:59244   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51952         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59623   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35399   ESTABLISHED
tcp        0      0 127.0.0.1:31000         127.0.0.1:52258         ESTABLISHED
tcp        0      0 148.251.187.175:59240   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52147         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:47186   ESTABLISHED
tcp        0      0 127.0.0.1:33761         127.0.0.1:16600         ESTABLISHED
tcp        0      0 148.251.187.175:36071   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:30100         127.0.0.1:42905         ESTABLISHED
tcp        0      0 127.0.0.1:52247         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51912         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52353         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52135         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35544   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59491   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:37326         127.0.0.1:20000         TIME_WAIT  
tcp        0      0 148.251.187.175:59616   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:43328   148.251.187.175:80      TIME_WAIT  
tcp        0      0 127.0.0.1:52058         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:43179   148.251.187.175:80      TIME_WAIT  
tcp        0      0 127.0.0.1:50048         127.0.0.1:16500         TIME_WAIT  
tcp        0      0 127.0.0.1:51916         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:50473         127.0.0.1:21000         TIME_WAIT  
tcp        0      0 127.0.0.1:51835         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51921         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59631   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59590   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:52336         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52349         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59225   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59343   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59394   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52164         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51905         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35450   ESTABLISHED
tcp        0      0 127.0.0.1:53187         127.0.0.1:17500         TIME_WAIT  
tcp        0      0 127.0.0.1:50324         127.0.0.1:16500         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35416   ESTABLISHED
tcp        0      0 127.0.0.1:51795         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:34928   ESTABLISHED
tcp        0      0 127.0.0.1:16600         127.0.0.1:33761         ESTABLISHED
tcp        0      0 127.0.0.1:49940         127.0.0.1:16500         TIME_WAIT  
tcp        0      0 148.251.187.175:59600   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59267   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52258         127.0.0.1:31000         ESTABLISHED
tcp        0      0 127.0.0.1:53295         127.0.0.1:17500         TIME_WAIT  
tcp        0      0 127.0.0.1:49357         127.0.0.1:31100         ESTABLISHED
tcp        0      0 127.0.0.1:52099         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:39399   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:34760   54.228.246.64:20000     TIME_WAIT  
tcp        0      0 127.0.0.1:51911         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35304   ESTABLISHED
tcp        0      0 148.251.187.175:44930   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:34697         127.0.0.1:27017         TIME_WAIT  
tcp        0      0 127.0.0.1:49836         127.0.0.1:16500         TIME_WAIT  
tcp        0      0 148.251.187.175:59238   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51932         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:38880   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59229   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51918         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59584   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51810         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59274   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52273         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:50355         127.0.0.1:21000         TIME_WAIT  
tcp        0      0 148.251.187.175:59227   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:52488   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59141   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:37326         127.0.0.1:31000         ESTABLISHED
tcp        0      0 148.251.187.175:59366   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59378   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52073         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:57982         127.0.0.1:30000         TIME_WAIT  
tcp        0      0 127.0.0.1:51922         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52275         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:54373   ESTABLISHED
tcp        0      0 127.0.0.1:57870         127.0.0.1:30000         TIME_WAIT  
tcp        0    311 148.251.187.175:57484   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:56128   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:59590   ESTABLISHED
tcp        0      1 148.251.187.175:54813   148.251.187.176:4949    SYN_SENT   
tcp        1      0 148.251.187.175:51058   148.251.187.175:27017   CLOSE_WAIT 
tcp        0      0 148.251.187.175:59253   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:34927   ESTABLISHED
tcp        0      0 148.251.187.175:59605   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35613   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:35319   ESTABLISHED
tcp        0      0 127.0.0.1:51861         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59509   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52024         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35802   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:51928         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52012         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:31000         127.0.0.1:51893         ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:35436   ESTABLISHED
tcp        0      0 148.251.187.175:59581   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:35408   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:39120         127.0.0.1:31000         ESTABLISHED
tcp        0      0 127.0.0.1:52184         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35486   ESTABLISHED
tcp        0      0 127.0.0.1:52078         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35802   ESTABLISHED
tcp        0      0 148.251.187.175:59139   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52027         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35356   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59387   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:51924         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59247   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59235   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:35385   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.175:59136   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:36057   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:35385   ESTABLISHED
tcp        0      0 148.251.187.175:27017   148.251.187.175:46380   ESTABLISHED
tcp        0      0 127.0.0.1:48176         127.0.0.1:4949          TIME_WAIT  
tcp        0      0 127.0.0.1:52279         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:51822         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52253         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:59780         127.0.0.1:31000         ESTABLISHED
tcp        0      0 148.251.187.175:59628   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:58996   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59127   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:59344   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.175:60793   5.45.99.167:22          ESTABLISHED
tcp        0      0 148.251.187.175:35302   148.251.187.175:27017   ESTABLISHED
tcp        0      0 127.0.0.1:32825         127.0.0.1:31100         ESTABLISHED
tcp        0      0 127.0.0.1:51989         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59388   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 127.0.0.1:52087         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52191         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 127.0.0.1:52192         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:35356   54.228.246.64:21000     TIME_WAIT  
tcp        0      0 148.251.187.175:27017   148.251.187.175:35521   ESTABLISHED
tcp        0      0 127.0.0.1:51783         127.0.0.1:31000         TIME_WAIT  
tcp        0      0 148.251.187.175:59351   148.251.187.175:27017   TIME_WAIT  
tcp6       0      0 :::80                   :::*                    LISTEN     
tcp6       0      0 :::9200                 :::*                    LISTEN     
tcp6       0      0 :::6002                 :::*                    LISTEN     
tcp6       0      0 :::9300                 :::*                    LISTEN     
tcp6       0      0 :::4949                 :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 :::25                   :::*                    LISTEN     
tcp6       0      0 :::443                  :::*                    LISTEN     
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57444   ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57440   ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57441   ESTABLISHED
tcp6       0      0 148.251.187.175:57444   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:57439   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:443     78.46.174.247:36461     ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57438   ESTABLISHED
tcp6       0      0 148.251.187.175:57436   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57446   ESTABLISHED
tcp6       0      0 148.251.187.175:57438   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:57440   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:57435   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:57445   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57443   ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57437   ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57435   ESTABLISHED
tcp6       0      0 148.251.187.175:57443   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:57437   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:57442   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57439   ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57442   ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57436   ESTABLISHED
tcp6       0      0 148.251.187.175:57434   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:57446   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57445   ESTABLISHED
tcp6       0      0 148.251.187.175:57441   148.251.187.175:9300    ESTABLISHED
tcp6       0      0 148.251.187.175:9300    148.251.187.175:57434   ESTABLISHED
udp        0      0 148.251.187.175:123     0.0.0.0:*                          
udp        0      0 127.0.0.1:123           0.0.0.0:*                          
udp        0      0 0.0.0.0:123             0.0.0.0:*                          
udp6       0      0 :::54328                :::*                               
udp6       0      0 fe80::ec4:7aff:fe07:123 :::*                               
udp6       0      0 ::1:123                 :::*                               
udp6       0      0 :::123                  :::*                               
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     48532011 /tmp/mongodb-27017.sock
unix  2      [ ACC ]     STREAM     LISTENING     5380609  public/pickup
unix  2      [ ACC ]     STREAM     LISTENING     5380613  public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     5380616  public/qmgr
unix  2      [ ACC ]     STREAM     LISTENING     5380620  private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     11541    /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     5380623  private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     5380626  private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     5380629  private/defer
unix  2      [ ACC ]     STREAM     LISTENING     5380632  private/trace
unix  2      [ ACC ]     STREAM     LISTENING     5380635  private/verify
unix  2      [ ACC ]     STREAM     LISTENING     5380638  public/flush
unix  2      [ ACC ]     STREAM     LISTENING     76409355 @/tmp/.X11-unix/X2
unix  2      [ ACC ]     STREAM     LISTENING     5380641  private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     5380644  private/proxywrite
unix  2      [ ACC ]     STREAM     LISTENING     5380647  private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     5380650  private/relay
unix  2      [ ACC ]     STREAM     LISTENING     5380653  public/showq
unix  2      [ ACC ]     STREAM     LISTENING     5380656  private/error
unix  2      [ ACC ]     STREAM     LISTENING     5380659  private/retry
unix  2      [ ACC ]     STREAM     LISTENING     5380662  private/discard
unix  2      [ ACC ]     STREAM     LISTENING     76409356 /tmp/.X11-unix/X2
unix  2      [ ACC ]     STREAM     LISTENING     5380665  private/local
unix  2      [ ACC ]     STREAM     LISTENING     5380668  private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     5380671  private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     5380674  private/anvil
unix  2      [ ACC ]     STREAM     LISTENING     5380677  private/scache
unix  2      [ ACC ]     STREAM     LISTENING     5380680  private/maildrop
unix  2      [ ACC ]     STREAM     LISTENING     5380683  private/uucp
unix  2      [ ACC ]     STREAM     LISTENING     5380686  private/ifmail
unix  2      [ ACC ]     STREAM     LISTENING     5380689  private/bsmtp
unix  2      [ ACC ]     STREAM     LISTENING     5380692  private/scalemail-backend
unix  2      [ ACC ]     STREAM     LISTENING     5380695  private/mailman
unix  2      [ ACC ]     STREAM     LISTENING     4668599  @/com/ubuntu/upstart
unix  2      [ ACC ]     SEQPACKET  LISTENING     7572     /run/udev/control
unix  23     [ ]         DGRAM                    13521    /dev/log
unix  2      [ ACC ]     STREAM     LISTENING     17617    /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     5380624  
unix  3      [ ]         STREAM     CONNECTED     5380697  
unix  3      [ ]         STREAM     CONNECTED     4668605  @/com/ubuntu/upstart
unix  3      [ ]         STREAM     CONNECTED     4631479  
unix  3      [ ]         STREAM     CONNECTED     5380643  
unix  3      [ ]         STREAM     CONNECTED     5380639  
unix  3      [ ]         STREAM     CONNECTED     5380693  
unix  3      [ ]         STREAM     CONNECTED     5380637  
unix  3      [ ]         STREAM     CONNECTED     5380691  
unix  3      [ ]         STREAM     CONNECTED     1349     
unix  3      [ ]         STREAM     CONNECTED     5380633  
unix  3      [ ]         STREAM     CONNECTED     5380685  
unix  3      [ ]         STREAM     CONNECTED     5380631  
unix  2      [ ]         DGRAM                    77368791 
unix  3      [ ]         STREAM     CONNECTED     5380618  
unix  3      [ ]         STREAM     CONNECTED     5380687  
unix  2      [ ]         DGRAM                    5376927  
unix  3      [ ]         STREAM     CONNECTED     14530    
unix  3      [ ]         STREAM     CONNECTED     5380611  
unix  3      [ ]         STREAM     CONNECTED     4668610  @/com/ubuntu/upstart
unix  3      [ ]         STREAM     CONNECTED     5380615  
unix  3      [ ]         STREAM     CONNECTED     5380681  
unix  3      [ ]         STREAM     CONNECTED     5380627  
unix  3      [ ]         STREAM     CONNECTED     5380614  
unix  3      [ ]         STREAM     CONNECTED     5380679  
unix  2      [ ]         DGRAM                    77363780 
unix  3      [ ]         STREAM     CONNECTED     5380661  
unix  3      [ ]         STREAM     CONNECTED     4668600  
unix  3      [ ]         STREAM     CONNECTED     5380608  
unix  3      [ ]         STREAM     CONNECTED     5380657  
unix  3      [ ]         STREAM     CONNECTED     4666216  
unix  3      [ ]         STREAM     CONNECTED     5380675  
unix  3      [ ]         STREAM     CONNECTED     5380655  
unix  2      [ ]         STREAM     CONNECTED     68405317 
unix  3      [ ]         STREAM     CONNECTED     5380673  
unix  3      [ ]         STREAM     CONNECTED     5380669  
unix  3      [ ]         STREAM     CONNECTED     5380667  
unix  3      [ ]         STREAM     CONNECTED     5380651  
unix  3      [ ]         STREAM     CONNECTED     5380649  
unix  2      [ ]         DGRAM                    77370390 
unix  3      [ ]         STREAM     CONNECTED     5380610  
unix  2      [ ]         DGRAM                    77363119 
unix  3      [ ]         STREAM     CONNECTED     5380663  
unix  3      [ ]         DGRAM                    15455    
unix  3      [ ]         STREAM     CONNECTED     5380645  
unix  3      [ ]         STREAM     CONNECTED     1348     
unix  3      [ ]         STREAM     CONNECTED     5380625  
unix  3      [ ]         STREAM     CONNECTED     5380642  
unix  3      [ ]         STREAM     CONNECTED     5380694  
unix  3      [ ]         STREAM     CONNECTED     5380640  
unix  3      [ ]         STREAM     CONNECTED     5380696  
unix  2      [ ]         DGRAM                    14625    
unix  3      [ ]         STREAM     CONNECTED     5380622  
unix  3      [ ]         STREAM     CONNECTED     5380690  
unix  3      [ ]         STREAM     CONNECTED     4620791  
unix  3      [ ]         STREAM     CONNECTED     5380636  
unix  2      [ ]         DGRAM                    5380585  
unix  3      [ ]         STREAM     CONNECTED     5380634  
unix  2      [ ]         STREAM     CONNECTED     68394910 
unix  3      [ ]         STREAM     CONNECTED     5380688  
unix  2      [ ]         DGRAM                    77342112 
unix  2      [ ]         DGRAM                    77363790 
unix  3      [ ]         STREAM     CONNECTED     5380621  
unix  2      [ ]         DGRAM                    77187874 
unix  2      [ ]         DGRAM                    77369616 
unix  2      [ ]         DGRAM                    77363125 
unix  3      [ ]         STREAM     CONNECTED     5380630  
unix  3      [ ]         STREAM     CONNECTED     5380684  
unix  3      [ ]         STREAM     CONNECTED     5380628  
unix  3      [ ]         STREAM     CONNECTED     16579    /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    13623    
unix  3      [ ]         STREAM     CONNECTED     5380682  
unix  3      [ ]         STREAM     CONNECTED     5380678  
unix  2      [ ]         DGRAM                    77366673 
unix  2      [ ]         DGRAM                    77367718 
unix  3      [ ]         STREAM     CONNECTED     5380676  
unix  3      [ ]         STREAM     CONNECTED     5380660  
unix  2      [ ]         DGRAM                    7703     
unix  3      [ ]         STREAM     CONNECTED     5380658  
unix  2      [ ]         DGRAM                    14511    
unix  2      [ ]         DGRAM                    76341332 
unix  3      [ ]         STREAM     CONNECTED     5380607  
unix  3      [ ]         STREAM     CONNECTED     4668604  @/com/ubuntu/upstart
unix  3      [ ]         STREAM     CONNECTED     5380672  
unix  2      [ ]         DGRAM                    5376882  
unix  3      [ ]         STREAM     CONNECTED     5380654  
unix  3      [ ]         STREAM     CONNECTED     5380670  
unix  3      [ ]         STREAM     CONNECTED     5380652  
unix  3      [ ]         STREAM     CONNECTED     5380617  
unix  3      [ ]         STREAM     CONNECTED     5380648  
unix  2      [ ]         DGRAM                    77367835 
unix  2      [ ]         DGRAM                    9719142  
unix  3      [ ]         STREAM     CONNECTED     5380666  
unix  3      [ ]         STREAM     CONNECTED     5380646  
unix  3      [ ]         STREAM     CONNECTED     4519817  /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     5380664  
unix  3      [ ]         DGRAM                    15454    
 
===
 
root@diesel2 /opt/mongo/bin # netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN     
tcp        0    216 148.251.187.176:22      78.46.174.247:60804     ESTABLISHED
tcp        0      0 127.0.0.1:38151         127.0.0.1:27017         TIME_WAIT  
tcp        0      0 148.251.187.176:27017   148.251.187.175:47667   ESTABLISHED
tcp6       0      0 :::22                   :::*                    LISTEN     
udp        0      0 148.251.187.176:123     0.0.0.0:*                          
udp        0      0 127.0.0.1:123           0.0.0.0:*                          
udp        0      0 0.0.0.0:123             0.0.0.0:*                          
udp6       0      0 2a01:4f8:211:928::2:123 :::*                               
udp6       0      0 fe80::ec4:7aff:fe07:123 :::*                               
udp6       0      0 ::1:123                 :::*                               
udp6       0      0 :::123                  :::*                               
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     411536   /tmp/mongodb-27017.sock
unix  2      [ ACC ]     STREAM     LISTENING     7778     /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     7534     @/com/ubuntu/upstart
unix  2      [ ACC ]     SEQPACKET  LISTENING     7576     /run/udev/control
unix  7      [ ]         DGRAM                    15530    /dev/log
unix  2      [ ACC ]     STREAM     LISTENING     11465    /var/run/acpid.socket
unix  3      [ ]         STREAM     CONNECTED     7981     @/com/ubuntu/upstart
unix  3      [ ]         STREAM     CONNECTED     15546    
unix  3      [ ]         STREAM     CONNECTED     7758     @/com/ubuntu/upstart
unix  3      [ ]         DGRAM                    15490    
unix  2      [ ]         DGRAM                    495828   
unix  2      [ ]         DGRAM                    15545    
unix  2      [ ]         DGRAM                    7900     
unix  3      [ ]         STREAM     CONNECTED     13359    
unix  3      [ ]         STREAM     CONNECTED     11438    /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     12511    @/com/ubuntu/upstart
unix  3      [ ]         STREAM     CONNECTED     15548    /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    7990     
unix  3      [ ]         DGRAM                    15489    
unix  3      [ ]         STREAM     CONNECTED     15547    
unix  3      [ ]         STREAM     CONNECTED     9458     
unix  3      [ ]         STREAM     CONNECTED     8299     
unix  2      [ ]         DGRAM                    7808     
unix  3      [ ]         STREAM     CONNECTED     1532     
unix  3      [ ]         STREAM     CONNECTED     7827  
 
====================
 
AFTER
 
root@diesel /opt/mongo/bin # netstat -na > out.txt
 
The output is too long to be printed here. 1.4 MB in total. For the most part it consists of thousands of lines like the one below. Perhaps this says anything to you. I'm no network expert, but to me it looks like it somehow cannot connect to 147.251.187.176:27017. But when I tried a telnet "147.251.187.176:27017" on the primary (even during the problem) it connects immediately!? I'm unsure. Perhaps I'm doing something very obvious wrong here - but for the life of me I just can't figure out what it might be...
 
tcp        0      0 148.251.187.175:53342   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:54081   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:49513   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:47464   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:50710   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:53713   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:55139   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:48835   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:54111   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:55914   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:47098   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:50965   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:47485   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:42297   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:43344   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:55425   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:56289   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:56607   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:51421   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:54377   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:41137   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:47649   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:50793   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:52807   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:54544   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:48638   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:53454   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:42100   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:46622   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:45901   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:50142   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:46870   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:41728   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:45312   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:51157   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:44340   148.251.187.176:27017   TIME_WAIT
tcp        0      0 148.251.187.175:52737   148.251.187.176:27017   TIME_WAIT
 
root@diesel2 /opt/mongo/bin # netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN     
tcp        0      0 148.251.187.176:27017   148.251.187.175:44874   ESTABLISHED
tcp        0      0 148.251.187.176:41922   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.176:27017   148.251.187.175:48247   ESTABLISHED
tcp        0      0 148.251.187.176:41920   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.176:41919   148.251.187.175:27017   ESTABLISHED
tcp        0     36 148.251.187.176:22      78.46.174.247:60821     ESTABLISHED
tcp        0      0 148.251.187.176:41923   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.176:22      78.46.174.247:60804     ESTABLISHED
tcp        0      0 148.251.187.176:27017   148.251.187.175:47961   ESTABLISHED
tcp        0      0 148.251.187.176:27017   148.251.187.175:47667   ESTABLISHED
tcp        0      0 148.251.187.176:41921   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.176:41906   148.251.187.175:27017   TIME_WAIT  
tcp        0      0 148.251.187.176:41921   148.251.187.175:27017   ESTABLISHED
tcp        0      0 148.251.187.176:41906   148.251.187.175:27017   TIME_WAIT  
tcp6       0      0 :::22                   :::*                    LISTEN     
udp        0      0 148.251.187.176:123     0.0.0.0:*                          
udp        0      0 127.0.0.1:123           0.0.0.0:*                          
udp        0      0 0.0.0.0:123             0.0.0.0:*                          
udp6       0      0 2a01:4f8:211:928::2:123 :::*                               
udp6       0      0 fe80::ec4:7aff:fe07:123 :::*                               
udp6       0      0 ::1:123                 :::*                               
udp6       0      0 :::123                  :::*                               
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     411536   /tmp/mongodb-27017.sock
unix  2      [ ACC ]     STREAM     LISTENING     7778     /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     7534     @/com/ubuntu/upstart
unix  2      [ ACC ]     SEQPACKET  LISTENING     7576     /run/udev/control
unix  8      [ ]         DGRAM                    15530    /dev/log
unix  2      [ ACC ]     STREAM     LISTENING     11465    /var/run/acpid.socket
unix  3      [ ]         STREAM     CONNECTED     7981     @/com/ubuntu/upstart
unix  3      [ ]         STREAM     CONNECTED     15546    
unix  3      [ ]         STREAM     CONNECTED     7758     @/com/ubuntu/upstart
unix  3      [ ]         DGRAM                    15490    
unix  2      [ ]         DGRAM                    495828   
unix  2      [ ]         DGRAM                    15545    
unix  2      [ ]         DGRAM                    7900     
unix  2      [ ]         DGRAM                    543669   
unix  3      [ ]         STREAM     CONNECTED     13359    
unix  3      [ ]         STREAM     CONNECTED     11438    /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     12511    @/com/ubuntu/upstart
unix  3      [ ]         STREAM     CONNECTED     15548    /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    7990     
unix  3      [ ]         DGRAM                    15489    
unix  3      [ ]         STREAM     CONNECTED     15547    
unix  3      [ ]         STREAM     CONNECTED     9458     
unix  3      [ ]         STREAM     CONNECTED     8299     
unix  2      [ ]         DGRAM                    7808     
unix  3      [ ]         STREAM     CONNECTED     1532     
unix  3      [ ]         STREAM     CONNECTED     7827   
 

Comment by Scott Hernandez (Inactive) [ 02/Feb/15 ]

It would be best to get those values from both servers during the problem. Getting the netstat output before and after would be helpful as well, for comparison.

How would you go about creating the problem again? If this will affect production I might be able to help mitigate any downtime if I see your exact steps.

Comment by rgpublic [ 02/Feb/15 ]

Yes. IP for diesel.kryd.int is correct. diesel2 is the same with .176 at the end.
Do you need the netstat output only for the primary or also for the secondary? And do I need to bring up the secondary again and recreate the problem or is a netstat right now sufficient when only the primary is running (As I said the system quickly becomes unusable if I bring up the secondary)?

Comment by Scott Hernandez (Inactive) [ 02/Feb/15 ]

The message just means that there was a problem writing on the network socket (in a specific place in code, which isn't terribly interesting).

As I expected this seems to point to a network issue, so let's move on to collect netstat info. Can you please run the following, and attach the output:

netstat -na

name ip
diesel.kryd.int 148.251.187.175
diesel2.kryd.int 148.251.187.176
Comment by rgpublic [ 02/Feb/15 ]

OK, It's a bit difficult to get all the information you requested, because this is a production system and when I try to do replication the server shuts down after a few minutes. And the primary is running for such a long time that I can't find the "startup part of the logs". Nevertheless I tried my best to get as much info as possible. It all begins with me starting the new server that should become secondary (I'm indeed starting as root which Mongo is complaining about, but I hope this is not such a severe problem, I've made sure max open files is set to 64000 as recommended):

2015-02-02T15:33:33.144+0100 I CONTROL  ***** SERVER RESTARTED *****
2015-02-02T15:33:33.147+0100 I CONTROL  [initandlisten] MongoDB starting : pid=7641 port=27017 dbpath=/var/lib/mongodb 64-bit host=diesel2
2015-02-02T15:33:33.147+0100 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-02-02T15:33:33.147+0100 I CONTROL  [initandlisten] 
2015-02-02T15:33:33.147+0100 I CONTROL  [initandlisten] db version v3.0.0-rc7
2015-02-02T15:33:33.147+0100 I CONTROL  [initandlisten] git version: e4c60053b2967e16f765fa25d16aa6d629faa196
2015-02-02T15:33:33.147+0100 I CONTROL  [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-02-02T15:33:33.147+0100 I CONTROL  [initandlisten] allocator: tcmalloc
2015-02-02T15:33:33.147+0100 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", replication: { replSet: "diesel" }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }
2015-02-02T15:33:33.164+0100 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-02-02T15:33:33.518+0100 I REPL     [initandlisten] Did not find local replica set configuration document at startup;  NoMatchingDocument Did not find replica set configuration document in local.system.replset
2015-02-02T15:33:33.551+0100 I NETWORK  [initandlisten] waiting for connections on port 27017

Then, back on the primary I run rs.conf() and rs.status();

diesel:PRIMARY> rs.conf();
{
        "_id" : "diesel",
        "version" : 94644,
        "members" : [
                {
                        "_id" : 1,
                        "host" : "diesel.kryd.int:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 2,
                        "tags" : {
 
                        },
                        "slaveDelay" : 0,
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "diesel2.kryd.int:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
 
                        },
                        "slaveDelay" : 0,
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatTimeoutSecs" : 10,
                "getLastErrorModes" : {
 
                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                }
        }
 
        
        
===
 
diesel:PRIMARY> rs.status();
{
        "set" : "diesel",
        "date" : ISODate("2015-02-02T14:36:27.419Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 1,
                        "name" : "diesel.kryd.int:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 615817,
                        "optime" : Timestamp(1422887787, 33),
                        "optimeDate" : ISODate("2015-02-02T14:36:27Z"),
                        "electionTime" : Timestamp(1422645983, 1),
                        "electionDate" : ISODate("2015-01-30T19:26:23Z"),
                        "configVersion" : 94644,
                        "self" : true
                },
                {
                        "_id" : 2,
                        "name" : "diesel2.kryd.int:27017",
                        "health" : 1,
                        "state" : 5,
                        "stateStr" : "STARTUP2",
                        "uptime" : 21,
                        "optime" : Timestamp(0, 0),
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2015-02-02T14:36:25.493Z"),
                        "lastHeartbeatRecv" : ISODate("2015-02-02T14:36:25.545Z"),
                        "pingMs" : 0,
                        "syncingTo" : "diesel.kryd.int:27017",
                        "configVersion" : 94644
                }
        ],
        "ok" : 1
}
 

Now, I run rs.add("diesel2.kryd.int") and then I can run rs.status() and rs.conf() on the secondary which is - IMHO correctly - in STARTUP2 phase:

diesel:STARTUP2> rs.conf();
{
        "_id" : "diesel",
        "version" : 94644,
        "members" : [
                {
                        "_id" : 1,
                        "host" : "diesel.kryd.int:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 2,
                        "tags" : {
 
                        },
                        "slaveDelay" : 0,
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "diesel2.kryd.int:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
 
                        },
                        "slaveDelay" : 0,
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatTimeoutSecs" : 10,
                "getLastErrorModes" : {
 
                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                }
        }
}
 
 
====
 
diesel:STARTUP2> rs.status();
{
        "set" : "diesel",
        "date" : ISODate("2015-02-02T14:37:05.278Z"),
        "myState" : 5,
        "syncingTo" : "diesel.kryd.int:27017",
        "members" : [
                {
                        "_id" : 1,
                        "name" : "diesel.kryd.int:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 59,
                        "optime" : Timestamp(1422887795, 49),
                        "optimeDate" : ISODate("2015-02-02T14:36:35Z"),
                        "lastHeartbeat" : ISODate("2015-02-02T14:37:03.560Z"),
                        "lastHeartbeatRecv" : ISODate("2015-02-02T14:37:04.117Z"),
                        "pingMs" : 0,
                        "electionTime" : Timestamp(1422645983, 1),
                        "electionDate" : ISODate("2015-01-30T19:26:23Z"),
                        "configVersion" : 94644
                },
                {
                        "_id" : 2,
                        "name" : "diesel2.kryd.int:27017",
                        "health" : 1,
                        "state" : 5,
                        "stateStr" : "STARTUP2",
                        "uptime" : 212,
                        "optime" : Timestamp(0, 0),
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "syncingTo" : "diesel.kryd.int:27017",
                        "configVersion" : 94644,
                        "self" : true
                }
        ],
        "ok" : 1
}

Now, where things get probably ugly is in the logs of the secondary. I found this:

2015-02-02T15:40:45.534+0100 I -        [rsSync] caught exception (socket exception [SEND_ERROR] for 148.251.187.175:27017) in destructor (~PiggyBackData)
2015-02-02T15:40:45.536+0100 E REPL     [rsSync] 16465 recv failed while exhausting cursor
2015-02-02T15:40:45.536+0100 E REPL     [rsSync] initial sync attempt failed, 9 attempts remaining
2015-02-02T15:40:45.636+0100 I REPL     [ReplicationExecutor] Member diesel.kryd.int:27017 is now in state SECONDARY
2015-02-02T15:40:46.207+0100 I REPL     [ReplicationExecutor] replSetElect voting yea for diesel.kryd.int:27017 (1)
2015-02-02T15:40:47.636+0100 I REPL     [ReplicationExecutor] Member diesel.kryd.int:27017 is now in state PRIMARY
2015-02-02T15:40:50.536+0100 I REPL     [rsSync] initial sync pending
2015-02-02T15:40:50.536+0100 I REPL     [ReplicationExecutor] syncing from: diesel.kryd.int:27017
2015-02-02T15:40:50.539+0100 I REPL     [rsSync] initial sync drop all databases
2015-02-02T15:40:50.539+0100 I STORAGE  [rsSync] dropAllDatabasesExceptLocal 2
2015-02-02T15:40:50.584+0100 I REPL     [rsSync] initial sync clone all databases
2015-02-02T15:40:50.589+0100 I REPL     [rsSync] initial sync cloning db: edison
2015-02-02T15:40:50.639+0100 I INDEX    [rsSync] build index on: edison.fileids properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "edison.fileids" }
2015-02-02T15:40:50.639+0100 I INDEX    [rsSync]         building index using bulk method
2015-02-02T15:40:50.680+0100 I INDEX    [rsSync] build index done.  scanned 109 total records. 0 secs
2015-02-02T15:41:04.455+0100 I QUERY    [conn1] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:config.settings query:{}
2015-02-02T15:41:50.122+0100 I STORAGE  [rsSync] clone edison.fs 1656703
2015-02-02T15:41:51.177+0100 I STORAGE  [rsSync] 1656967 objects cloned so far from collection edison.fs
2015-02-02T15:41:59.432+0100 I QUERY    [conn2] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:config.settings query:{}
2015-02-02T15:42:50.434+0100 I STORAGE  [rsSync] clone edison.fs 1908735
2015-02-02T15:42:52.252+0100 I STORAGE  [rsSync] 1909189 objects cloned so far from collection edison.fs
2015-02-02T15:42:54.448+0100 I QUERY    [conn1] assertion 13436 not master or secondary; cannot currently read from this replSet member ns:config.settings query:{}
2015-02-02T15:43:32.271+0100 I -        [rsSync] caught exception (socket exception [SEND_ERROR] for 148.251.187.175:27017) in destructor (~PiggyBackData)
2015-02-02T15:43:32.273+0100 E REPL     [rsSync] 16465 recv failed while exhausting cursor
2015-02-02T15:43:32.273+0100 E REPL     [rsSync] initial sync attempt failed, 8 attempts remaining

The "8 attempts remaining" now ticks down to 0 and the primary becomes secondary and everything is offline, because there is no primary anymore

I suspect the problem is this:

socket exception [SEND_ERROR] for 148.251.187.175:27017) in destructor (~PiggyBackData)

but I don't know what this is supposed to mean...

Comment by Scott Hernandez (Inactive) [ 01/Feb/15 ]

This error message is coming from the operating system.

Can you please attach:

  • rs.conf(), rs.status() from each member
  • operating system information from all server in the replica set
  • the startup part of the logs from all servers
  • the (repeated) error message sections

Generally errors like these, which are really coming from the operating system, stem from some environment or resource contention/limitation. So another step is to try to restart that member and see if starting from a clean boot results in the same error or new behavior...

After we get the requested information we can start to collect additional things if that is not sufficient. In particular the state of tcp sockets, and the number of them between the member may be a clear indication of the underlying operating system issue.

Comment by rgpublic [ 31/Jan/15 ]

The logs are huge and contain sensitive queries because the primary is under active use. Anything special you need and I could search/grep?

Comment by Ramon Fernandez Marina [ 31/Jan/15 ]

rgpublic, can you please upload logs for both servers?

Generated at Thu Feb 08 03:43:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.