[SERVER-500] bind() fails in test Created: 21/Dec/09  Updated: 12/Jul/16  Resolved: 09/Feb/10

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: None
Fix Version/s: 1.3.2

Type: Bug Priority: Major - P3
Reporter: Aaron Staple Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

m33058| Mon Dec 21 13:21:52 connection accepted from 127.0.0.1:59544 #1
shell: started mongo program /home/buildslave/buildslave/Linux_32bit/mongo/mongod --nohttpinterface --noprealloc --bind_ip 127.0.0.1 --smallfiles --port 54675 --dbpath /data/db/repl6tests-1-slave --slave --source 127.0.0.1:33058 --oplogSize 1
m54675| Mon Dec 21 13:21:52 Mongo DB : starting : pid = 2105 port = 54675 dbpath = /data/db/repl6tests-1-slave master = 0 slave = 1 32-bit
m54675|
m54675| ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
m54675| ** see http://blog.mongodb.org/post/137788967/32-bit-limitations for more
m54675|
m54675| Mon Dec 21 13:21:52 db version v1.3.0-, pdfile version 4.5
m54675| Mon Dec 21 13:21:52 git version: 499cb647b65ca9d1b0cbdc27f72359b30bf2bf39
m54675| Mon Dec 21 13:21:52 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37
m54675| Mon Dec 21 13:21:52 waiting for connections on port 54675
m54675| Mon Dec 21 13:21:53 connection accepted from 127.0.0.1:33904 #1
shell: started mongo program /home/buildslave/buildslave/Linux_32bit/mongo/mongod --nohttpinterface --noprealloc --bind_ip 127.0.0.1 --smallfiles --port 59544 --dbpath /data/db/repl6tests-2-slave --slave --source 127.0.0.1:33058 --oplogSize 1
m59544| Mon Dec 21 13:21:53 Mongo DB : starting : pid = 2117 port = 59544 dbpath = /data/db/repl6tests-2-slave master = 0 slave = 1 32-bit
m59544|
m59544| ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
m59544| ** see http://blog.mongodb.org/post/137788967/32-bit-limitations for more
m59544|
m59544| Mon Dec 21 13:21:53 db version v1.3.0-, pdfile version 4.5
m59544| Mon Dec 21 13:21:53 git version: 499cb647b65ca9d1b0cbdc27f72359b30bf2bf39
m59544| Mon Dec 21 13:21:53 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37
m59544| Mon Dec 21 13:21:53 waiting for connections on port 59544
m59544| Mon Dec 21 13:21:53 listen(): bind() failed errno:98 Address already in use

allocatePorts allocates, opens, then closes (a) port(s). Our assumption is that the os will not utilize the same ports again soon, but in this failure mode one of the allocated ports is reused for a Mongo client before the mongod server for which the port was allocated is started. Either the system is low on available ports, or the port allocation algorithm does not attempt to avoid reuse on short timescales.

Potential solutions:
1) port allocator could leave port open, then consumer of an allocated port would close it immediately before re-opening it as desired
2) all mongod server instances are started before all mongo client instances - not practical for many of the more complicated repl/pairing tests



 Comments   
Comment by auto [ 24/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 clean up processes in test
http://github.com/mongodb/mongo/commit/a05b907fc4b4641a7d135df735b95cee386d5b27

Comment by auto [ 24/Feb/10 ]

Author:

{'login': 'erh', 'name': 'Eliot Horowitz', 'email': 'eliot@10gen.com'}

Message: SERVER-500 - use ports < 32867 and no http interface for sharding tests
http://github.com/mongodb/mongo/commit/fcb6a16b48324629122c7da28d75b3e8393fd913

Comment by auto [ 24/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 use lower port for scons mongod
http://github.com/mongodb/mongo/commit/074553298d403babb7265de7900257bb550b9ce7

Comment by auto [ 24/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 lower base port for e2e tests
http://github.com/mongodb/mongo/commit/92b27770bec81ead7f360d0feb6e8bf8e8d83fbc

Comment by auto [ 24/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 try hardcoded ports
http://github.com/mongodb/mongo/commit/c62aa841a22a391c9e1c9223955970cea9ec074f

Comment by Aaron Staple [ 09/Feb/10 ]

Should be done, go ahead and close when you're ready.

Comment by Eliot Horowitz (Inactive) [ 09/Feb/10 ]

resolving because it seems to be done
just assigning it back to you (aaron) in case i missed something you wanted to do

Comment by auto [ 09/Feb/10 ]

Author:

{'login': 'erh', 'name': 'Eliot Horowitz', 'email': 'eliot@10gen.com'}

Message: SERVER-500 - use ports < 32867 and no http interface for sharding tests
http://github.com/mongodb/mongo/commit/5eb5dab79fa6f24fe0a1b6739c510ac3a0f46e32

Comment by Aaron Staple [ 09/Feb/10 ]

Sorry, where I said "ethereal range" above it should have been "ephemeral range"

Comment by Aaron Staple [ 09/Feb/10 ]

We shouldn't have the sharding tests use ports with the 39999 base. Also it would be nice if we disabled the http interfaces for the sharding test mongods

Comment by auto [ 09/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 use lower port for scons mongod
http://github.com/mongodb/mongo/commit/1e7d97e82c8e8a512bc164c8f895b1f671a85466

Comment by auto [ 09/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 remove debugging
http://github.com/mongodb/mongo/commit/fcc79ef58ed578dd3d245e8edbbaad772f559e32

Comment by auto [ 09/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 lower base port for e2e tests
http://github.com/mongodb/mongo/commit/fd8f4be646e9d4811aefcc3671d7a0c3d7f828e4

Comment by Aaron Staple [ 09/Feb/10 ]

The problem appears to be that the bridge is connecting to itself. Apparently if you try to connect to a port in the ethereal range and that port happens to be allocated for the socket as it attempts to connect, the socket will connect to itself. On my linux vm the ethereal range begins at 32768, so I'm going to change the base port for the repl tests from 35000 to 31000. Hopefully the buildbot machines have similar ethereal port ranges, we'll see. If this works we may want to lower all testing ports below the 32768 threshold (for sharding tests, scons tests, etc).

Comment by auto [ 08/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 debug
http://github.com/mongodb/mongo/commit/16b0080200a86e337eaaea06f4e7fdf949efdfa1

Comment by auto [ 08/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 debugging
http://github.com/mongodb/mongo/commit/9be0100c844caf68705dd8e4b904175e83875849

Comment by Eliot Horowitz (Inactive) [ 07/Feb/10 ]

making it hard to get releases out, etc.. sometimes

Comment by auto [ 03/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 improve netstat output
http://github.com/mongodb/mongo/commit/194afe347e0bce5ee61f54a1bfd4f46850ecece3

Comment by Aaron Staple [ 03/Feb/10 ]

<http://buildbot.mongodb.org/builders/Linux%2064-bit%20Legacy/builds/75/steps/test_2/logs/stdio>

Comment by auto [ 02/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 more info about allocated sockets
http://github.com/mongodb/mongo/commit/36fd4db154a925d45bc1f03392e232ef101c37b0

Comment by auto [ 02/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 some linux distros require -atp syntax
http://github.com/mongodb/mongo/commit/b16ad764e52ed055cf7098b5d624026378429f57

Comment by Aaron Staple [ 02/Feb/10 ]

bind() sometimes fails even in the absence of a client using the desired port. I've added some debugging to investigate, possibly the ports are used by a process outside the test harness.

Comment by auto [ 02/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 debug port usage when running tests
http://github.com/mongodb/mongo/commit/68909a6d715e57f61039f4ceb450bca9127fdc93

Comment by auto [ 02/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 clean up processes in test
http://github.com/mongodb/mongo/commit/857518458441d8182669c115575f341f05a0a6f1

Comment by auto [ 02/Feb/10 ]

Author:

{'login': 'astaple', 'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 try hardcoded ports
http://github.com/mongodb/mongo/commit/a0010982f7d252f082391d85a1c7f13c7b1a0f54

Comment by Aaron Staple [ 19/Jan/10 ]

<http://buildbot.mongodb.org/builders/Linux%2064-bit/builds/607/steps/test_2/logs/stdio>

Comment by Aaron Staple [ 12/Jan/10 ]

<http://buildbot.mongodb.org/builders/Linux%2032-bit/builds/554/steps/test_2/logs/stdio>

Comment by Aaron Staple [ 12/Jan/10 ]

The new port allocation strategy I tried doesn't work well on linux. I'm reverting for now.

Comment by auto [ 12/Jan/10 ]

Author:

{'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: Revert "SERVER-500 after allocating ports, keep sockets open until ports will be used; use udp sockets"

This reverts commit 1430432a39e2f1c9c8b937e9ef4124640b9cf00e.
http://github.com/mongodb/mongo/commit/29f41538ed7a148e5434b30550dc80c9a1466733

Comment by auto [ 12/Jan/10 ]

Author:

{'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: Revert "SERVER-500 MINOR handle hardcoded ports as well"

This reverts commit 2460cc0e2d4a2e09bc8f140bc22f4b8a8f442667.
http://github.com/mongodb/mongo/commit/ca537582dd1682ee5e0de2176701b006bda3ada6

Comment by auto [ 12/Jan/10 ]

Author:

{'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: Revert "SERVER-500 more port/socket corner cases"

This reverts commit b691dc629e0696de36ba7b7607f4fd7c28a8e86b.
http://github.com/mongodb/mongo/commit/4c03735ef8a4d5eb2bdced759e2e87b146bde0f0

Comment by auto [ 11/Jan/10 ]

Author:

{'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 more port/socket corner cases
http://github.com/mongodb/mongo/commit/b691dc629e0696de36ba7b7607f4fd7c28a8e86b

Comment by auto [ 11/Jan/10 ]

Author:

{'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 MINOR handle hardcoded ports as well
http://github.com/mongodb/mongo/commit/2460cc0e2d4a2e09bc8f140bc22f4b8a8f442667

Comment by Aaron Staple [ 11/Jan/10 ]

hopefully fixed w/ above commit

Comment by auto [ 11/Jan/10 ]

Author:

{'name': 'Aaron', 'email': 'aaron@10gen.com'}

Message: SERVER-500 after allocating ports, keep sockets open until ports will be used; use udp sockets
http://github.com/mongodb/mongo/commit/1430432a39e2f1c9c8b937e9ef4124640b9cf00e

Comment by Aaron Staple [ 11/Jan/10 ]

<http://buildbot.mongodb.org/builders/Linux%2032-bit/builds/537/steps/test_2/logs/stdio>

Comment by Aaron Staple [ 21/Dec/09 ]

<http://buildbot.mongodb.org/builders/Linux%2032-bit/builds/431/steps/test_2/logs/stdio>

Generated at Thu Feb 08 02:54:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.