[SERVER-6591] Localhost authentication exception doesn't work right on sharded cluster Created: 25/Jul/12 Updated: 27/Oct/15 Resolved: 20/Nov/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Security, Sharding |
| Affects Version/s: | 2.2.0-rc0 |
| Fix Version/s: | 2.2.4, 2.3.1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Spencer Brody (Inactive) | Assignee: | Spencer Brody (Inactive) |
| Resolution: | Done | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
If you have a sharded cluster all running on one machine, in 2.0 if you connect to the mongos via localhost and there are no admin users then it allows you full access to the mongos. Since the connections between mongos and mongod have full access anyway, this gives you full access to the cluster. In 2.1.2+, however, authentication for commands is done on the mongods, with the credentials passed from mongos. Some machines will not consider the connection from the mongos to the mongods to be a localhost connection if the cluster was configured using the machine's hostname. This means that even though you connect to mongos on a local connection, some commands might still fail. On the other hand, some machines DO recognize the connection between mongos and mongod as a localhost connection. On those machines if you add an admin user to the cluster, which should close the localhost backdoor, commands that are passed through to the mongods directly can still succeed, even without write authorization. In order to disable the localhost exception completely, you need to add admin users to each shard directly. This only affects clusters that are all running on the same machine, so it's not really a security hole, it's more a problem for our test infrastructure because it makes the behavior of authentication in tests vary based on which machine the tests are run on, and whether or not the connections between the mongos and mongods get considered local or not (seems to be related to whether the hostname for itself on the machine resolves to 127.0.0.1, or to the machine's public IP address). |
| Comments |
| Comment by auto [ 23/Mar/13 ] |
|
Author: {u'date': u'2013-03-23T19:03:05Z', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}Message: |
| Comment by auto [ 27/Feb/13 ] |
|
Author: {u'date': u'2013-02-27T19:08:06Z', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}Message: |
| Comment by auto [ 27/Feb/13 ] |
|
Author: {u'date': u'2013-02-11T23:06:33Z', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}Message: |
| Comment by auto [ 27/Feb/13 ] |
|
Author: {u'date': u'2013-02-11T18:28:07Z', u'name': u'Craig Wilson', u'email': u'craiggwilson@gmail.com'}Message: |
| Comment by auto [ 27/Feb/13 ] |
|
Author: {u'date': u'2012-11-20T15:47:02Z', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}Message: |
| Comment by auto [ 27/Feb/13 ] |
|
Author: {u'date': u'2012-11-17T01:44:21Z', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}Message: |
| Comment by auto [ 11/Feb/13 ] |
|
Author: {u'date': u'2013-02-11T23:06:33Z', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}Message: |
| Comment by auto [ 11/Feb/13 ] |
|
Author: {u'date': u'2013-02-11T18:34:42Z', u'name': u'Spencer Brody', u'email': u'spencer@10gen.com'}Message: Merge pull request #374 from craiggwilson/server6591
|
| Comment by auto [ 11/Feb/13 ] |
|
Author: {u'date': u'2013-02-11T18:28:07Z', u'name': u'Craig Wilson', u'email': u'craiggwilson@gmail.com'}Message: |
| Comment by auto [ 20/Nov/12 ] |
|
Author: {u'date': u'2012-11-20T15:47:02Z', u'email': u'spencer@10gen.com', u'name': u'Spencer T Brody'}Message: |
| Comment by auto [ 20/Nov/12 ] |
|
Author: {u'date': u'2012-11-17T01:44:21Z', u'email': u'spencer@10gen.com', u'name': u'Spencer T Brody'}Message: |
| Comment by auto [ 20/Nov/12 ] |
|
Author: {u'date': u'2012-11-20T14:43:01Z', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}Message: Revert " This reverts commit d6fb51e8b1b224c90a025723bab6e605385ae338. |
| Comment by auto [ 20/Nov/12 ] |
|
Author: {u'date': u'2012-11-17T01:44:21Z', u'email': u'spencer@10gen.com', u'name': u'Spencer T Brody'}Message: |
| Comment by Spencer Brody (Inactive) [ 13/Nov/12 ] |
|
Hi Ian, |
| Comment by Ian Patton [ 13/Nov/12 ] |
|
Now that we have clarified that this affects production clusters and not just test setups can we please prioritize a fix higher than "2.3 desired"? This is clearly a regression. |
| Comment by Spencer Brody (Inactive) [ 13/Nov/12 ] |
|
Just the config servers and a mongos should be sufficient to add the first user to the admin database. |
| Comment by Ian Patton [ 13/Nov/12 ] |
|
Eliot, your documentation now points to this ticket: When setting up authentication for the first time you must either:
New in version 2.0: Support for authentication with sharded clusters. Before 2.0 sharded clusters had to run with trusted applications and a trusted networking configuration. Consider the Control Access to MongoDB Instances with Authentication document which outlines procedures for configuring and maintaining users and access with MongoDB’s authentication system. [1] Because of So what I experienced is exactly what is now documented. You cannot add a user to the admin database when starting a sharded cluster with --keyfile in MongoDB 2.2. What is the minimal set of mongo servers in my cluster that I need to start without the keyfile to get a user added? Just the 3 config servers and a MongoS? Or do any of the shards need to be up? |
| Comment by Eliot Horowitz (Inactive) [ 13/Nov/12 ] |
|
Ian - this is really meant only for test clusters where all nodes are on the same host. |
| Comment by Ian Patton [ 13/Nov/12 ] |
|
I just got burned by this during a 2am redeployment. I had been planning to add authentication to our automated deployment process since mongo 2.0 but just got it done. Now I find out things do not work the way they were documented, and the fix is not even scheduled. This kind of thing is very very frustrating. I would prefer that things work the way they are intended before you add a single new feature. Please schedule this for a fix in 2.2.2 or 2.2.3 and not make us live with this for a year. |
| Comment by Spencer Brody (Inactive) [ 05/Sep/12 ] |
|
@Zoiner, the easiest workaround for adding the first user to an authenticated cluster is to add the user with auth disabled on the whole cluster, then restart the cluster enabling auth. |
| Comment by Zoiner Tejada [ 05/Sep/12 ] |
|
So is there a workaround for enabling authentication at all on a distributed sharded cluster? See my reference architecture in Server-6936. If we want to automate setup, but are fine adding admins during setup is there a correct order in which we can enable security and use admin users instead of relying on the localhost authentication? For example, something like add the users to each member of the shard, then to the configs prior to starting routers? I'm reaching here, I know, but trying to find a workaround that doesn't involve waiting for 2.3 |
| Comment by Spencer Brody (Inactive) [ 05/Sep/12 ] |
|
This also means that if you have a sharded cluster where the shards are not running on the same machine as the mongos, then even if you connect to the mongos over localhost, you will still have commands that run on the shards or config servers get denied. |
| Comment by Spencer Brody (Inactive) [ 26/Jul/12 ] |
|
To fix this, you could have the mongos send the internalSecurityAuthenticationTable with all commands when the client connection is a localhost connection. You'd also have to do something different for the other side of this: closing the localhost exception after adding the first admin user when the shards/config servers DO recognize the connections from mongos as local. To do that you could make the localhost exception not apply to any connections authorized as __system. |
| Comment by Spencer Brody (Inactive) [ 26/Jul/12 ] |
|
This also means that if you create a ShardingTest with useHostname:true, you cannot add the first admin user through the mongos, because the config servers won't allow the addUser to succeed because they will think the connection from the mongos is not local. |
| Comment by Spencer Brody (Inactive) [ 25/Jul/12 ] |
|
Yes. Setting useHostname to false makes it configure the whole set using "localhost" instead of the machine's hostname, this makes it recognize the connections between mongos and the mongos as local (with the problems described above that commands run on the shards can then succeed even after adding an admin user to the cluster). If useHostname is false, then some machines resolve the hostname to 127.0.0.1 and consider the connections to be local, whereas others resolve it to the machine's public IP, and don't. |
| Comment by Randolph Tan [ 25/Jul/12 ] |
|
Have you tried using the useHostname option (for ShardingTest, ReplSetTest, MongoRunner.runMongod, etc)? |