[SERVER-6591] Localhost authentication exception doesn't work right on sharded cluster Created: 25/Jul/12  Updated: 27/Oct/15  Resolved: 20/Nov/12

Status: Closed
Project: Core Server
Component/s: Security, Sharding
Affects Version/s: 2.2.0-rc0
Fix Version/s: 2.2.4, 2.3.1

Type: Bug Priority: Major - P3
Reporter: Spencer Brody (Inactive) Assignee: Spencer Brody (Inactive)
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by DOCS-993 Update documents related to localhost... Closed
Duplicate
is duplicated by SERVER-6936 addUser appears to fail when KeyFile ... Closed
Related
Operating System: ALL
Participants:

 Description   

If you have a sharded cluster all running on one machine, in 2.0 if you connect to the mongos via localhost and there are no admin users then it allows you full access to the mongos. Since the connections between mongos and mongod have full access anyway, this gives you full access to the cluster.

In 2.1.2+, however, authentication for commands is done on the mongods, with the credentials passed from mongos. Some machines will not consider the connection from the mongos to the mongods to be a localhost connection if the cluster was configured using the machine's hostname. This means that even though you connect to mongos on a local connection, some commands might still fail.

On the other hand, some machines DO recognize the connection between mongos and mongod as a localhost connection. On those machines if you add an admin user to the cluster, which should close the localhost backdoor, commands that are passed through to the mongods directly can still succeed, even without write authorization. In order to disable the localhost exception completely, you need to add admin users to each shard directly.

This only affects clusters that are all running on the same machine, so it's not really a security hole, it's more a problem for our test infrastructure because it makes the behavior of authentication in tests vary based on which machine the tests are run on, and whether or not the connections between the mongos and mongods get considered local or not (seems to be related to whether the hostname for itself on the machine resolves to 127.0.0.1, or to the machine's public IP address).



 Comments   
Comment by auto [ 23/Mar/13 ]

Author:

{u'date': u'2013-03-23T19:03:05Z', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-6591 Visual Studio – adapt to renamed and added files
Branch: v2.2
https://github.com/mongodb/mongo/commit/8e252ca206118f3b64ccef4e857e2e84c26a10e0

Comment by auto [ 27/Feb/13 ]

Author:

{u'date': u'2013-02-27T19:08:06Z', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: SERVER-6591 Fix localhostAuthBypass tests to work on 2.2, where GLE requires auth to run
Branch: v2.2
https://github.com/mongodb/mongo/commit/cd6d81d6db45da1df6ad5b357d23d7b1e2f514ca

Comment by auto [ 27/Feb/13 ]

Author:

{u'date': u'2013-02-11T23:06:33Z', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: SERVER-6591 Fix test
Branch: v2.2
https://github.com/mongodb/mongo/commit/206bd9372623375aee7bd1c67520d93f96508105

Comment by auto [ 27/Feb/13 ]

Author:

{u'date': u'2013-02-11T18:28:07Z', u'name': u'Craig Wilson', u'email': u'craiggwilson@gmail.com'}

Message: SERVER-6591: added tests for bypassing authentication when connected with a localhost to a server that has no admin users.
Branch: v2.2
https://github.com/mongodb/mongo/commit/8cadee62dea7b2db8acb07e9eb5e6bb87f18d9c8

Comment by auto [ 27/Feb/13 ]

Author:

{u'date': u'2012-11-20T15:47:02Z', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: SERVER-6591 Don't query config server when running with noauth
Branch: v2.2
https://github.com/mongodb/mongo/commit/57ffb2b9993859213f6bc7cb8e664548377f6445

Comment by auto [ 27/Feb/13 ]

Author:

{u'date': u'2012-11-17T01:44:21Z', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: SERVER-6591 Fix localhost auth exception in sharded systems
Branch: v2.2
https://github.com/mongodb/mongo/commit/b4e33ce1ba54c6bfa4ff9ae0bd144d674e11c883

Comment by auto [ 11/Feb/13 ]

Author:

{u'date': u'2013-02-11T23:06:33Z', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: SERVER-6591 Fix test
Branch: master
https://github.com/mongodb/mongo/commit/331ca28913a345af97ddd1bca5bcfe423f40789e

Comment by auto [ 11/Feb/13 ]

Author:

{u'date': u'2013-02-11T18:34:42Z', u'name': u'Spencer Brody', u'email': u'spencer@10gen.com'}

Message: Merge pull request #374 from craiggwilson/server6591

SERVER-6591: added tests for bypassing authentication when connected wit...
Branch: master
https://github.com/mongodb/mongo/commit/918799b280d9920bc916f2e28e6dea37a3c86bfc

Comment by auto [ 11/Feb/13 ]

Author:

{u'date': u'2013-02-11T18:28:07Z', u'name': u'Craig Wilson', u'email': u'craiggwilson@gmail.com'}

Message: SERVER-6591: added tests for bypassing authentication when connected with a localhost to a server that has no admin users.
Branch: master
https://github.com/mongodb/mongo/commit/ae5f92aec64a2e65091cb53c4edfd77a12abacbf

Comment by auto [ 20/Nov/12 ]

Author:

{u'date': u'2012-11-20T15:47:02Z', u'email': u'spencer@10gen.com', u'name': u'Spencer T Brody'}

Message: SERVER-6591 Don't query config server when running with noauth
Branch: master
https://github.com/mongodb/mongo/commit/dac93da27163d2c1e0978a44ec814caaa80c5428

Comment by auto [ 20/Nov/12 ]

Author:

{u'date': u'2012-11-17T01:44:21Z', u'email': u'spencer@10gen.com', u'name': u'Spencer T Brody'}

Message: SERVER-6591 Fix localhost auth exception in sharded systems
Branch: master
https://github.com/mongodb/mongo/commit/85abee65027409182c318d22aa79fd224e6b311c

Comment by auto [ 20/Nov/12 ]

Author:

{u'date': u'2012-11-20T14:43:01Z', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}

Message: Revert "SERVER-6591 Fix localhost auth exception in sharded systems"

This reverts commit d6fb51e8b1b224c90a025723bab6e605385ae338.
Branch: master
https://github.com/mongodb/mongo/commit/675132c08da314433157016c3a0ee1187b55b65a

Comment by auto [ 20/Nov/12 ]

Author:

{u'date': u'2012-11-17T01:44:21Z', u'email': u'spencer@10gen.com', u'name': u'Spencer T Brody'}

Message: SERVER-6591 Fix localhost auth exception in sharded systems
Branch: master
https://github.com/mongodb/mongo/commit/d6fb51e8b1b224c90a025723bab6e605385ae338

Comment by Spencer Brody (Inactive) [ 13/Nov/12 ]

Hi Ian,
I just updated the fix version on the ticket. It will be fixed on the 2.2 series in 2.2.3, and will be fixed for the 2.4 release as well.

Comment by Ian Patton [ 13/Nov/12 ]

Now that we have clarified that this affects production clusters and not just test setups can we please prioritize a fix higher than "2.3 desired"? This is clearly a regression.

Comment by Spencer Brody (Inactive) [ 13/Nov/12 ]

Just the config servers and a mongos should be sufficient to add the first user to the admin database.

Comment by Ian Patton [ 13/Nov/12 ]

Eliot, your documentation now points to this ticket:
http://docs.mongodb.org/manual/administration/security/

When setting up authentication for the first time you must either:

  • add at least one user to the admin database before starting the mongod instance with auth.
  • add the first user to the admin database when connected to the mongod instance from a localhost connection. [1]

New in version 2.0: Support for authentication with sharded clusters. Before 2.0 sharded clusters had to run with trusted applications and a trusted networking configuration.

Consider the Control Access to MongoDB Instances with Authentication document which outlines procedures for configuring and maintaining users and access with MongoDB’s authentication system.

[1] Because of SERVER-6591, you cannot add the first user to a sharded cluster using the localhost connection in 2.2. If you are running a 2.2 sharded cluster, and want to enable authentication, you must deploy the cluster and add the first user to the admin database before restarting the cluster to run with keyFile.

So what I experienced is exactly what is now documented. You cannot add a user to the admin database when starting a sharded cluster with --keyfile in MongoDB 2.2.

What is the minimal set of mongo servers in my cluster that I need to start without the keyfile to get a user added? Just the 3 config servers and a MongoS? Or do any of the shards need to be up?

Comment by Eliot Horowitz (Inactive) [ 13/Nov/12 ]

Ian - this is really meant only for test clusters where all nodes are on the same host.
What happened in your case?

Comment by Ian Patton [ 13/Nov/12 ]

I just got burned by this during a 2am redeployment. I had been planning to add authentication to our automated deployment process since mongo 2.0 but just got it done. Now I find out things do not work the way they were documented, and the fix is not even scheduled. This kind of thing is very very frustrating.

I would prefer that things work the way they are intended before you add a single new feature. Please schedule this for a fix in 2.2.2 or 2.2.3 and not make us live with this for a year.

Comment by Spencer Brody (Inactive) [ 05/Sep/12 ]

@Zoiner, the easiest workaround for adding the first user to an authenticated cluster is to add the user with auth disabled on the whole cluster, then restart the cluster enabling auth.

Comment by Zoiner Tejada [ 05/Sep/12 ]

So is there a workaround for enabling authentication at all on a distributed sharded cluster? See my reference architecture in Server-6936. If we want to automate setup, but are fine adding admins during setup is there a correct order in which we can enable security and use admin users instead of relying on the localhost authentication? For example, something like add the users to each member of the shard, then to the configs prior to starting routers? I'm reaching here, I know, but trying to find a workaround that doesn't involve waiting for 2.3

Comment by Spencer Brody (Inactive) [ 05/Sep/12 ]

This also means that if you have a sharded cluster where the shards are not running on the same machine as the mongos, then even if you connect to the mongos over localhost, you will still have commands that run on the shards or config servers get denied.

Comment by Spencer Brody (Inactive) [ 26/Jul/12 ]

To fix this, you could have the mongos send the internalSecurityAuthenticationTable with all commands when the client connection is a localhost connection.

You'd also have to do something different for the other side of this: closing the localhost exception after adding the first admin user when the shards/config servers DO recognize the connections from mongos as local. To do that you could make the localhost exception not apply to any connections authorized as __system.

Comment by Spencer Brody (Inactive) [ 26/Jul/12 ]

This also means that if you create a ShardingTest with useHostname:true, you cannot add the first admin user through the mongos, because the config servers won't allow the addUser to succeed because they will think the connection from the mongos is not local.

Comment by Spencer Brody (Inactive) [ 25/Jul/12 ]

Yes. Setting useHostname to false makes it configure the whole set using "localhost" instead of the machine's hostname, this makes it recognize the connections between mongos and the mongos as local (with the problems described above that commands run on the shards can then succeed even after adding an admin user to the cluster).

If useHostname is false, then some machines resolve the hostname to 127.0.0.1 and consider the connections to be local, whereas others resolve it to the machine's public IP, and don't.

Comment by Randolph Tan [ 25/Jul/12 ]

Have you tried using the useHostname option (for ShardingTest, ReplSetTest, MongoRunner.runMongod, etc)?

Generated at Thu Feb 08 03:12:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.