[SERVER-14170] Cannot read from secondary if both audit and auth are enabled in a sharded cluster Created: 05/Jun/14  Updated: 11/Mar/15  Resolved: 17/Jun/14

Status: Closed
Project: Core Server
Component/s: Security
Affects Version/s: 2.6.0, 2.6.2
Fix Version/s: 2.6.4, 2.7.2

Type: Bug Priority: Critical - P2
Reporter: Linda Qin Assignee: Andy Schwerin
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
Tested
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Participants:

 Description   
Issue Status as of Jul 18, 2014

ISSUE SUMMARY
When both audit and authentication are enabled in a sharded cluster it is not possible to read from secondaries.

USER IMPACT
Attempts to read from a secondary fail with an error.

WORKAROUNDS
Users can read from primary nodes instead.

AFFECTED VERSIONS
MongoDB production releases in the 2.6 series up to 2.6.3 (inclusive) are affected by this issue.

FIX VERSION
The fix is included in the 2.6.4 production release.

RESOLUTION DETAILS
Do not invoke the runCommand hook when executing authentication commands. Executing this hook leads to commands that cannot be run by unauthenticated users, which prevents authentication of connections when auditing is enabled.

Original description
  1. Start a sharded cluster with audit and authentication enabled using MongoDB 2.6 enterprise.
  2. Create a user in the admin database with root role.
  3. Authenticate with this user.
  4. Insert a document to test database.
  5. Set read preference to secondary and it will fail to query the document.
  6. Set read preference to primary and it will return the document.

mongos> db.auth("root","root")
1
mongos> use test
switched to db test
mongos> db.docs.insert({x:1})
WriteResult({ "nInserted" : 1 })
mongos> db.getMongo().setReadPref('secondary' );
mongos> db.docs.findOne()
2014-06-04T22:26:36.647-0400 error: { "$err" : "not authorized for query on test.docs", "code" : 13 } at src/mongo/shell/query.js:131
mongos> db.getMongo().setReadPref('primary' );
mongos> db.docs.findOne()
{ "_id" : ObjectId("538fd55871f795733df60f5d"), "x" : 1 }



 Comments   
Comment by Githook User [ 17/Jun/14 ]

Author:

{u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@mongodb.com'}

Message: JS Regression test of SERVER-14170.
Branch: v2.6
https://github.com/10gen/mongo-enterprise-modules/commit/92428525e6368ee6307ad56a658e60bdaa0c3e07

Comment by Githook User [ 17/Jun/14 ]

Author:

{u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@mongodb.com'}

Message: SERVER-14170 Ensure that inserted document propagates to secondary before counting in test.

The test audit_read_from_sharded_secondaries.js inserts a document into a collection,
and then calls count on a secondary, expecting to count it. For this to succeed, the
insert needs a write concern high enough to ensure it reaches the secondary, in this
case w: 2 suffices.
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/7239defaa0af1a67ca388f2cfb7a4ecafec0036c

Comment by Githook User [ 17/Jun/14 ]

Author:

{u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@mongodb.com'}

Message: SERVER-14170 Do not invoke the RunCommandHook when executing authentication commands.

Executing this hook leads to commands that cannot be run by unauthenticated users,
which prevents authentication of connections when auditing is enabled.
Branch: v2.6
https://github.com/mongodb/mongo/commit/61f2f525ea3c8e5fea19a58b97e339f0d39f98cc

Comment by Andy Schwerin [ 17/Jun/14 ]

Reopening because of test failure due to insufficient write concern. CR forthcoming.

Comment by Githook User [ 10/Jun/14 ]

Author:

{u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@mongodb.com'}

Message: JS Regression test of SERVER-14170
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/346f8bb2f05bc107d228e869e39c86cb8c94e5bf

Comment by Githook User [ 10/Jun/14 ]

Author:

{u'username': u'andy10gen', u'name': u'Andy Schwerin', u'email': u'schwerin@mongodb.com'}

Message: SERVER-14170 Do not invoke the RunCommandHook when executing authentication commands.

Executing this hook leads to commands that cannot be run by unauthenticated users,
which prevents authentication of connections when auditing is enabled.
Branch: master
https://github.com/mongodb/mongo/commit/85461da82e7aca4285173ab7427cd468bd0f0fc2

Comment by Andy Schwerin [ 06/Jun/14 ]

The proximate symptom is that mongos is sending the impersonatedUsers argument on getnonce and the other three authentication commands, when it talks to secondaries. I suspect it also sends it on ismaster to secondaries. The root cause has to do with when we set the hook on connections to add the impersonatedUsers field, in relation to when we authenticate connections. milkie is thinking about possible solutions.

Comment by Matt Dannenberg [ 05/Jun/14 ]

Sharding appears to be a necessary condition. I did the above steps without any mongos or configdb and did not come up against the problem.

Comment by Matt Dannenberg [ 05/Jun/14 ]

reproed:

  1. start config db:

    ./mongod --configsvr --dbpath /data/config --port 27030 --auditDestination console --keyFile jstests/libs/key2 

  2. start mongos:

     ./mongos --configdb dannenstation.local:27030 --auditDestination console --port 27040 --keyFile jstests/libs/key2 

  3. start replset:

    ./mongod --replSet sh0 --dbpath /data/sh0 --port 27050 --auditDestination console --keyFile jstests/libs/key2 
    ./mongod --replSet sh0 --dbpath /data/sh1 --port 27051 --auditDestination console --keyFile jstests/libs/key2
    > conf = {_id: "sh0", members: [{host: "dannenstation.local:27050", _id:0}, {host: "dannenstation.local:27051", _id:1}]}
    > rs.initiate(conf) 

  4. addshard:

    mongos> sh.addShard( "sh0/dannenstation.local:27050,dannenstation.local:27051" ) 

  5. create user:

    mongos> use admin
    mongos> db.addUser({user: "root", pwd:"root", roles:["root"]}) 

  6. steps we were handed:

    mongos> db.auth("root", "root")
    1
    mongos> use test
    switched to db test
    mongos> db.docs.insert({x:1})
    WriteResult({ "nInserted" : 1 })
    mongos> db.getMongo().setReadPref('secondary' );
    mongos> db.docs.findOne()
    2014-06-05T06:40:27.729-0400 Error: error: { "$err" : "not authorized for query on test.docs", "code" : 13 }
        at Error (<anonymous>)
        at DBQuery.next (src/mongo/shell/query.js:133:15)
        at DBCollection.findOne (src/mongo/shell/collection.js:187:22)
        at (shell):1:9 at src/mongo/shell/query.js:133
    mongos>  db.getMongo().setReadPref('primary' );
    mongos> db.docs.findOne()
    { "_id" : ObjectId("539049098c49061b6749ae4c"), "x" : 1 }

Comment by Eric Milkie [ 05/Jun/14 ]

Hi Linda.
I failed to reproduce this with my first attempt.
Can you show me the command line you used to start the shardsvr (mongod) and the configsvr?
Also can you post the exact commands you used to create the first user? I want to compare with what I used.

Generated at Thu Feb 08 03:34:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.