[SERVER-48516] at startup, confirm replica set node with auth can connect to itself Created: 01/Jun/20 Updated: 29/Oct/23 Resolved: 13/Nov/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.4.0-rc7 |
| Fix Version/s: | 4.9.0, 4.4.4 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Mark Callaghan (Inactive) | Assignee: | Mark Benvenuto |
| Resolution: | Fixed | Votes: | 3 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||
| Backport Requested: |
v4.4
|
||||||||||||||||||||
| Sprint: | Execution Team 2020-06-15, Execution Team 2020-06-29, Security 2020-11-16 | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||
| Description |
|
I have a workload that uses a single-node replicaset and does a create index. It looks like I now must start the single-node with the --keyFile option for create index to work – see SERVER-48344. So my questions are:
This is new in 4.4.0-rc7 and I assume is from the fix for SERVER-48235. I will attach files, but below some info has been inlined From mongo.log
From currentOp()
The create index thread stack
|
| Comments |
| Comment by Githook User [ 11/Jan/21 ] | |||||||||||||||||||||||||||
|
Author: {'name': 'Mark Benvenuto', 'email': 'mark.benvenuto@mongodb.com', 'username': 'markbenvenuto'}Message: (cherry picked from commit c38ec5727899cd563791d5ea4ec054cf6322498c) | |||||||||||||||||||||||||||
| Comment by Louis Williams [ 04/Jan/21 ] | |||||||||||||||||||||||||||
|
michael.gorodilov@gmail.com, can you confirm whether the workaround in my previous comment solves your problem? I have requested a backport to 4.4 so that this problem may be fixed in the next release. | |||||||||||||||||||||||||||
| Comment by Mchl Grdlv [ 04/Jan/21 ] | |||||||||||||||||||||||||||
|
I have installed in synology docker Rocket.chat and mongo. Everything works well. "2021-01-04 08:49:57,stdout,"{\"t\": {\"$date\":\"2021-01-04T11:49:57.752+03:00\"},\"s\":\"I\", \"c\":\"STORAGE\", \"id\":3856202, \"ctx\":\"IndexBuildsCoordinatorMongod-2\",\"msg\":\"'voteCommitIndexBuild' command failed.\",\"attr\":{\"indexBuildUUID\":{\"uuid\":{\"$uuid\":\"d90def3b-4dc1-4cce-aa19-53c3f7eb094e\"}},\"responseStatus\":{\"operationTime\":{\"$timestamp\":{\"t\":1609750196,\"i\":1}},\"ok\":0.0,\"errmsg\":\"command voteCommitIndexBuild requires authentication\",\"code\":13,\"codeName\":\"Unauthorized\",\"$clusterTime\":{\"clusterTime\":{\"$timestamp\":{\"t\":1609750196,\"i\":1}},\"signature\":{\"hash\":{\"$binary\":{\"base64\":\"RtULaapbdm/5ooRgGl6jfYeyu4k=\",\"subType\":\"0\"}},\"keyId\":6890507829308817412}}}}} ,\"s\":\"I\", \"c\":\"STORAGE\", \"id\":3856202, \"ctx\":\"IndexBuildsCoordinatorMongod-1\",\"msg\":\"'voteCommitIndexBuild' command failed.\",\"attr\":{\"indexBuildUUID\":{\"uuid\":{\"$uuid\":\"5038dec8-d17f-49a7-afe8-896d37876686\"}},\"responseStatus\":{\"operationTime\":{\"$timestamp\":{\"t\":1609750196,\"i\":1}},\"ok\":0.0,\"errmsg\":\"command voteCommitIndexBuild requires authentication\",\"code\":13,\"codeName\":\"Unauthorized\",\"$clusterTime\":{\"clusterTime\":{\"$timestamp\":{\"t\":1609750196,\"i\":1}},\"signature\":{\"hash\":{\"$binary\":{\"base64\":\"RtULaapbdm/5ooRgGl6jfYeyu4k=\",\"subType\":\"0\"}},\"keyId\":6890507829308817412}}}}} ,\"s\":\"I\", \"c\":\"STORAGE\", \"id\":3856202, \"ctx\":\"IndexBuildsCoordinatorMongod-0\",\"msg\":\"'voteCommitIndexBuild' command failed.\",\"attr\":{\"indexBuildUUID\":{\"uuid\":{\"$uuid\":\"7b61f52d-9904-4dc8-9b06-aedb8fcadfad\"}},\"responseStatus\":{\"operationTime\":{\"$timestamp\":{\"t\":1609750196,\"i\":1}},\"ok\":0.0,\"errmsg\":\"command voteCommitIndexBuild requires authentication\",\"code\":13,\"codeName\":\"Unauthorized\",\"$clusterTime\":{\"clusterTime\":{\"$timestamp\":{\"t\":1609750196,\"i\":1}},\"signature\":{\"hash\":{\"$binary\":{\"base64\":\"RtULaapbdm/5ooRgGl6jfYeyu4k=\",\"subType\":\"0\"}},\"keyId\":6890507829308817412}}}}} ,\"s\":\"I\", \"c\":\"STORAGE\", \"id\":3856202, \"ctx\":\"IndexBuildsCoordinatorMongod-2\",\"msg\":\"'voteCommitIndexBuild' command failed.\",\"attr\":{\"indexBuildUUID\":{\"uuid\":{\"$uuid\":\"d90def3b-4dc1-4cce-aa19-53c3f7eb094e\"}},\"responseStatus\":{\"operationTime\":{\"$timestamp\":{\"t\":1609750186,\"i\":2}},\"ok\":0.0,\"errmsg\":\"command voteCommitIndexBuild requires authentication\",\"code\":13,\"codeName\":\"Unauthorized\",\"$clusterTime\":{\"clusterTime\":{\"$timestamp\":{\"t\":1609750186,\"i\":2}},\"signature\":{\"hash\":{\"$binary\":{\"base64\":\"2TFFt43KcKYAPhR+rXwog/vqyOI=\",\"subType\":\"0\"}},\"keyId\":6890507829308817412}}}}} ,\"s\":\"I\", \"c\":\"STORAGE\", \"id\":3856202, \"ctx\":\"IndexBuildsCoordinatorMongod-1\",\"msg\":\"'voteCommitIndexBuild' command failed.\",\"attr\":{\"indexBuildUUID\":{\"uuid\":{\"$uuid\":\"5038dec8-d17f-49a7-afe8-896d37876686\"}},\"responseStatus\":{\"operationTime\":{\"$timestamp\":{\"t\":1609750186,\"i\":2}},\"ok\":0.0,\"errmsg\":\"command voteCommitIndexBuild requires authentication\",\"code\":13,\"codeName\":\"Unauthorized\",\"$clusterTime\":{\"clusterTime\":{\"$timestamp\":{\"t\":1609750186,\"i\":2}},\"signature\":{\"hash\":{\"$binary\":{\"base64\":\"2TFFt43KcKYAPhR+rXwog/vqyOI=\",\"subType\":\"0\"}},\"keyId\":6890507829308817412}}}}} ,\"s\":\"I\", \"c\":\"STORAGE\", \"id\":3856202, \"ctx\":\"IndexBuildsCoordinatorMongod-0\",\"msg\":\"'voteCommitIndexBuild' command failed.\",\"attr\":{\"indexBuildUUID\":{\"uuid\":{\"$uuid\":\"7b61f52d-9904-4dc8-9b06-aedb8fcadfad\"}},\"responseStatus\":{\"operationTime\":{\"$timestamp\":{\"t\":1609750186,\"i\":2}},\"ok\":0.0,\"errmsg\":\"command voteCommitIndexBuild requires authentication\",\"code\":13,\"codeName\":\"Unauthorized\",\"$clusterTime\":{\"clusterTime\":{\"$timestamp\":{\"t\":1609750186,\"i\":2}},\"signature\":{\"hash\":{\"$binary\":{\"base64\":\"2TFFt43KcKYAPhR+rXwog/vqyOI=\",\"subType\":\"0\"}},\"keyId\":6890507829308817412}}}}} | |||||||||||||||||||||||||||
| Comment by Githook User [ 13/Nov/20 ] | |||||||||||||||||||||||||||
|
Author: {'name': 'Mark Benvenuto', 'email': 'mark.benvenuto@mongodb.com', 'username': 'markbenvenuto'}Message: | |||||||||||||||||||||||||||
| Comment by Louis Williams [ 27/Oct/20 ] | |||||||||||||||||||||||||||
|
Hi earthquake_90@mail.ru, if you are running into this issue, you will need to follow the documentation here that explains how to set up keyfile authentication within a replica set. The problem is that the mongod is unable to authenticate with itself. For a quick workaround, pass a --keyFile to the mongod, just like in Mark's previous comment:
Or in your mongod.conf:
| |||||||||||||||||||||||||||
| Comment by Ivan Strelnikov [ 27/Oct/20 ] | |||||||||||||||||||||||||||
|
Hi! any updates? | |||||||||||||||||||||||||||
| Comment by Eric Milkie [ 09/Jul/20 ] | |||||||||||||||||||||||||||
|
I changed the description to match the suggested work to solve this ticket. Alternatively, we can modify AsyncDBClient to know when it is talking to itself and to switch to a different internal DBClient that doesn't use the network. | |||||||||||||||||||||||||||
| Comment by Eric Milkie [ 09/Jul/20 ] | |||||||||||||||||||||||||||
|
Technical details: Using a replica set with auth enabled but without a way for nodes to authenticate to one another is not a supported configuration, but there is currently nothing in the code that warns you about this situation if you only have one node up in your replica set. Note that other operations besides createIndexes would identify this problem; for example, adding a non-voting node to a one-node replica set would seem to succeed, and yet nothing would replicate to the new node. There is no active indication anything is wrong in this situation other than error messages appearing in the system log (and the lack of new data on the newly added node). This problem is not new to 4.4 and has existed since probably 2.2 and prior. The change in behavior for createIndexes, however, is new in 4.4. To make this problem more noticeable for createIndexes, we cannot fix the problem when the node goes to voteCommitIndexBuild, as at that point it is too late to abort the index build. Instead, what we can do is add a check on startup that a replicaset node can talk to itself, and abort initialization if this check fails, with an explanatory message. This will prevent surprise behaviors like this in the future, since there exists a wide variety of failure cases when running a single-voting-node replica set with an incorrect authentication configuration, and it would be difficult to document all of them and keep such a list up to date for each release. | |||||||||||||||||||||||||||
| Comment by Mark Callaghan (Inactive) [ 01/Jun/20 ] | |||||||||||||||||||||||||||
|
ixbug.tar has PMP stack traces, mongo.log, db.currentOp() output and the output from "ps" for mongod I use this mongo.conf
Previously I started mongod via bin/mongod --config mongo.conf Now I use
|