[SERVER-11776] Replication 'isself' check should allow mapped ports Created: 19/Nov/13  Updated: 04/Apr/17  Resolved: 29/May/14

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.4.10, 2.6.1
Fix Version/s: 2.6.4, 2.7.2

Type: Bug Priority: Major - P3
Reporter: Rajat Chopra Assignee: Scott Hernandez (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-14078 Verify >1 local interface works with ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Participants:

 Description   

'isself' should match the ports only if the hosts are being string matched. In cases where an instance is being addressed through a proxy port, port matching is an incorrect check - it invalidates a genuine match.

e.g. a mongod instance listening on port 27017 may be get redirected traffic from another port through IPtables routing. Such instances would fail to get added to a replica set for no good reason. Port proxies maybe required in multi-tenant cloud based systems where bind-able IPs/ports are NAT'ed.

Take this example replica set config to setup a one node cluster. The server itself can be addressed using a DNS name and an externally proxied port, but we cannot form a cluster if the same is used for the rs config.

config = { 
                "id"             : "test", 
                "members" : [ { 
                                         "_id" : 0, 
                                         "host" : "public_dns_to_ec2_node:38132" 
                                        } ] 
                }

Internally, the mongod was bound to 127.0.0.1:27017 and we have IPtables proxy rule to route all tcp traffic from 38132 to 127.0.0.1:27017, e.g.

-A OUTPUT -d 10.203.77.175/32 -m tcp -p tcp --dport 38132 -j DNAT --to-destination 127.0.0.1:27017
-A PREROUTING -d 10.203.77.175/32 -m tcp -p tcp --dport 38132 -j DNAT --to-destination 127.0.0.1:27017

where 'public_dns_to_ec2_node' routes to 10.203.77.175 through public IP DNS registration.

With the above example it is clear that a fix is needed to not unnecessarily match the ports as integers. Or we cannot even create a one node cluster, leave aside a usable 3-node replica set.



 Comments   
Comment by Githook User [ 17/Jul/14 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-11776 move port check outside loop
(cherry picked from commit da2701673900d4a0e70aca04fb9a34f424469601)
Branch: v2.6
https://github.com/mongodb/mongo/commit/aa09de7fd38af00ec9f7a4b689392809adaa2072

Comment by Githook User [ 17/Jul/14 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-11776 added scott's replication test for the isself command with mapped ports
(cherry picked from commit 8f4148960e4107beb3314d7cd3214066f82c1d99)
Branch: v2.6
https://github.com/mongodb/mongo/commit/34c561dbe12b0e6ea5b4f3ebe61aebee0021d6d2

Comment by Githook User [ 17/Jul/14 ]

Author:

{u'username': u'rajatchopra', u'name': u'Rajat Chopra', u'email': u'rchopra@redhat.com'}

Message: SERVER-11776 Replication 'isself' check should allow mapped ports

'isself' should match the ports only if the hosts are being string matched.
In cases where an instance is being addressed through a proxy port,
port matching is an incorrect check - it invalidates a genuine match.

Signed-off-by: Benety Goh <benety@mongodb.com>
(cherry picked from commit 4bb1a6155bc7bfb7cb69ff6844d427f0159a888c)
Branch: v2.6
https://github.com/mongodb/mongo/commit/6a8c3c8631e0d9850eaff83985731e47d7016185

Comment by Githook User [ 29/May/14 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-11776 move port check outside loop
Branch: master
https://github.com/mongodb/mongo/commit/da2701673900d4a0e70aca04fb9a34f424469601

Comment by Githook User [ 29/May/14 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-11776 added scott's replication test for the isself command with mapped ports
Branch: master
https://github.com/mongodb/mongo/commit/8f4148960e4107beb3314d7cd3214066f82c1d99

Comment by Githook User [ 29/May/14 ]

Author:

{u'username': u'rajatchopra', u'name': u'Rajat Chopra', u'email': u'rchopra@redhat.com'}

Message: SERVER-11776 Replication 'isself' check should allow mapped ports

'isself' should match the ports only if the hosts are being string matched.
In cases where an instance is being addressed through a proxy port,
port matching is an incorrect check - it invalidates a genuine match.

Signed-off-by: Benety Goh <benety@mongodb.com>
Branch: master
https://github.com/mongodb/mongo/commit/4bb1a6155bc7bfb7cb69ff6844d427f0159a888c

Comment by Scott Hernandez (Inactive) [ 30/Apr/14 ]

rajatchopra, we are out of the 2.6 dev/test/release cycle and are ready to commit this but we need a test to go with it first. The test will have to show that having a differently configured port on a replica still allows it to find itself at startup. We can simulate this in our javascript testing framework using mongobridge so the ports are different in the configuration and listening mongod.

I'm going to schedule this work for the next development release (2.7.2 to be specific). After that work we can look at the behavior under more use-cases and testing to decide if we can backport this behavior into 2.6, or earlier versions.

Comment by Scott Hernandez (Inactive) [ 20/Nov/13 ]

Thank you for the code contribution and investigation.

We will have to review the code and run the patch through our tests before we can respond again. That will probably happen next week.

Comment by Rajat Chopra [ 20/Nov/13 ]

@scott - in the given scenario the host can indeed be reached by the client using 'public_dns:38132', as well as with "127.0.0.1:27017". As a use case, I would refer to docker.io containers running mongodb -
https://groups.google.com/forum/#!topic/docker-user/WrLzky9GDV4
See the second comment in the forum link above, and without the improvement suggested here, a real replica set seems unattainable.

Solution proposed in pull request - https://github.com/mongodb/mongo/pull/543

Comment by Scott Hernandez (Inactive) [ 19/Nov/13 ]

You must make the connection work from itseft too, not just externally. Have you added a rule to redirect the port locally? Can you connect to "public_dns_to_ec2_node:38132" from the mongo client from that host?

All members in the replica set must be able to connect to each other. Also, please remember that you can run more than one member on the same host so the port, and unique identification, is fairly important here.

Generated at Thu Feb 08 03:26:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.