[SERVER-25064] Auth regression in nightlies on Windows and Solaris Created: 14/Jul/16  Updated: 19/Jul/16  Resolved: 17/Jul/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 3.3.10

Type: Bug Priority: Major - P3
Reporter: A. Jesse Jiryu Davis Assignee: Kaloian Manassiev
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Sharding 17 (07/15/16), Sharding 18 (08/05/16)
Participants:

 Description   

Beginning a few days ago, the mongo shell cannot authenticate to mongos during the C Driver's Evergreen startup. For an example task failure, see https://evergreen.mongodb.com/task/mongo_c_driver_windows_64_vs2010_integration_test_latest_sharded_7e0105c26ef1b295739465afc8d4dac6218f0ea5_16_07_14_17_31_13.

It logs:

 [2016/07/13 22:01:21.860] connecting to: 127.0.0.1:27017/admin
 [2016/07/13 22:01:21.862] MongoDB server version: 3.3.9-222-g9e7a989
 [2016/07/13 22:01:21.873] 2016-07-14T02:01:21.873+0000 E QUERY    [thread1] Error: Authentication failed. :
 [2016/07/13 22:01:21.873] DB.prototype._authOrThrow@src/mongo/shell/db.js:1414:20
 [2016/07/13 22:01:21.873] @(auth):6:1
 [2016/07/13 22:01:21.873] @(auth):1:2
 [2016/07/13 22:01:21.878] exception: login failed



 Comments   
Comment by Githook User [ 17/Jul/16 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-25064 Make mongos upgrade w:1 to w:majority for auth writes

Also extend the auth tests to validate that both variants work.
Branch: master
https://github.com/mongodb/mongo/commit/ff441b6607acaa551d8508af71b84a657ea66161

Comment by Spencer Jackson [ 14/Jul/16 ]

This stack trace from that log makes me suspect the problem occurs when trying to create the user:

[2016/07/14 13:57:48.656] curl -s --data @"$ORCHESTRATION_FILE" "$ORCHESTRATION_URL"
[2016/07/14 13:57:52.947] ./mongodb/bin/mongo $MONGO_SHELL_CONNECTION_FLAGS --eval 'printjson(db.serverBuildInfo())' admin
[2016/07/14 13:57:52.973] {"version": "0.9", "links": [{"href": "/v1/releases", "method": "GET", "rel": "get-releases"}, {"href": "/v1", "method": "GET", "rel": "self"}, {"href": "/v1/servers", "method": "GET", "rel": "get-servers"}, {"href": "/v1/servers", "method": "POST", "rel": "add-server"}, {"href": "/v1/replica_sets", "method": "POST", "rel": "add-replica-set"}, {"href": "/v1/replica_sets", "method": "GET", "rel": "get-replica-sets"}, {"href": "/v1/sharded_clusters", "method": "POST", "rel": "add-sharded-cluster"}, {"href": "/v1/sharded_clusters", "method": "GET", "rel": "get-sharded-clusters"}], "service": "mongo-orchestration"}["Traceback (most recent call last):\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\mongo_orchestration\\apps\\__init__.py\", line 62, in wrap\n    return f(*arg, **kwd)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\mongo_orchestration\\apps\\sharded_clusters.py\", line 68, in sh_create\n    result = _sh_create(data)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\mongo_orchestration\\apps\\sharded_clusters.py\", line 44, in _sh_create\n    cluster_id = ShardedClusters().create(params)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\mongo_orchestration\\sharded_clusters.py\", line 460, in create\n    cluster = ShardedCluster(params)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\mongo_orchestration\\sharded_clusters.py\", line 130, in __init__\n    fsync=True)))\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\mongo_orchestration\\common.py\", line 125, in _add_users\n    db.add_user(**secondary_login)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\pymongo\\database.py\", line 903, in add_user\n    (not uinfo[\"users\"]), name, password, read_only, **kwargs)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\pymongo\\database.py\", line 826, in _create_or_update_user\n    self.command(command_name, name, **opts)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\pymongo\\database.py\", line 481, in command\n    codec_options, **kwargs)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\pymongo\\database.py\", line 393, in _command\n    allowable_errors)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\pymongo\\pool.py\", line 239, in command\n    read_concern)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\pymongo\\network.py\", line 102, in command\n    helpers._check_command_response(response_doc, None, allowable_errors)\n", "  File \"c:\\data\\mci\\f255b0d60cb6da50287cd975227c409e\\mongo-c-driver\\venv\\lib\\site-packages\\pymongo\\helpers.py\", line 205, in _check_command_response\n    raise OperationFailure(msg % errmsg, code, response)\n", "OperationFailure: Invalid replication write concern.  Writes to config server replica sets must use w:'majority', got: { w: 1, fsync: true, wtimeout: 0 }\n"]2016-07-14T17:57:52.972+0000 I CONTROL  [main] Hotfix KB2731284 or later update is not installed, will zero-out data files

That error,

OperationFailure: Invalid replication write concern. Writes to config server replica sets must use w:'majority', got: { w: 1, fsync: true, wtimeout: 0 }

gets emitted from ShardingCatalogClientImpl::runUserManagementWriteCommand if `!writeConcern.validForConfigServers()`.

7c67f3a6fd52ea4933c23ce89f1cc5da62628ec4, in the range identified by max.hirschhorn, seems to have changed the behavior of validForConfigServers. kaloian.manassiev, can you take a look at this? Does it make sense that these errors should be observed, and do you have any idea why they might only be manifesting on specific platforms?

Comment by Max Hirschhorn [ 14/Jul/16 ]

Based on the last success and the first failure, the change in behavior of the server occurred between 3.3.9-58-g4efc4c9 and 3.3.9-199-g7991001.

Generated at Thu Feb 08 04:08:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.