[SERVER-18614] mongodb keyFile between replicas throws Permission denied Created: 21/May/15  Updated: 21/May/15  Resolved: 21/May/15

Status: Closed
Project: Core Server
Component/s: Admin, Replication, Security
Affects Version/s: 3.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Juan Manuel Diego G Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:
  1. Setup single node replica set with auth enabled, a root user and a keyFile
  2. rs.initiate() in this first node
  3. In the same server, setup a second mongod process in a different port with the same keyFile and replica set name
  4. Populate some data into the first proccess that should be PRIMARY and the only node of the replica
  5. From the shell of the first process, after db.auth(), try to rs.add() the second node
  6. It will fail, reason: Permission denied to connect to the second process from the first one
Participants:

 Description   

I have a single node ReplicaSet with auth activated, a root user and a keyFile I've created with this tutorial, I also have two more mongod processes in the same server in different ports (37017 and 47017) and the same replSet name, but when I try to add the secondaries in the mongo shell connected to PRIMARY with rs.add("172.31.48.41:37017") I get:

{
    "ok" : 0,
    "errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: 172.31.48.41:27017; the following nodes did not respond affirmatively: 172.31.48.41:37017 failed with Failed attempt to connect to 172.31.48.41:37017; couldn't connect to server 172.31.48.41:37017 (172.31.48.41), connection attempt failed",
    "code" : 74
}

Then I went to the mongod process log of the PRIMARY and found out this:

2015-05-19T20:53:59.848-0400 I REPL     [conn51] replSetReconfig admin command received from client
2015-05-19T20:53:59.848-0400 W NETWORK  [conn51] Failed to connect to 172.31.48.41:37017, reason: errno:13 Permission denied
2015-05-19T20:53:59.848-0400 I REPL     [conn51] replSetReconfig config object with 2 members parses ok
2015-05-19T20:53:59.849-0400 W NETWORK  [ReplExecNetThread-0] Failed to connect to 172.31.48.41:37017, reason: errno:13 Permission denied
2015-05-19T20:53:59.849-0400 W REPL     [ReplicationExecutor] Failed to complete heartbeat request to 172.31.48.41:37017; Location18915 Failed attempt to connect to 172.31.48.41:37017; couldn't connect to server 172.31.48.41:37017 (172.31.48.41), connection attempt failed
2015-05-19T20:53:59.849-0400 E REPL     [conn51] replSetReconfig failed; NodeNotFound Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: 172.31.48.41:27017; the following nodes did not respond affirmatively: 172.31.48.41:37017 failed with Failed attempt to connect to 172.31.48.41:37017; couldn't connect to server 172.31.48.41:37017 (172.31.48.41), connection attempt failed

And the log of the mongod that should become SECONDARY shows nothing, the last two lines are:

2015-05-19T20:48:36.584-0400 I REPL     [initandlisten] Did not find local replica set configuration document at startup;  NoMatchingDocument Did not find replica set configuration document in local.system.replset
2015-05-19T20:48:36.591-0400 I NETWORK  [initandlisten] waiting for connections on port 37017

It's clear that I cannot rs.initiate() in this node because it will self vote to be PRIMARY and that would create a conflict, so the line that states "Did not find local replica set configuration document at startup" is to be ignores as far as I know.

So I would think that the permission should be ok since I'm using the same key file in every mongod process and the replSet is the same in every config file, and that's all the tutorial states to be needed, but obviously something is missing.

Any ideas? Is this a bug?



 Comments   
Comment by Ramon Fernandez Marina [ 21/May/15 ]

The replica set is configured to talk to 172.31.48.41, but accodring to netstat there's no mongod process listening there.

Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group.

Regards,
Ramón.

Comment by Juan Manuel Diego G [ 21/May/15 ]

I'm have an open shell to 37017, so...

netstat | grep 37017 gives me:

tcp        0      0 localhost:37017         localhost:53124         ESTABLISHED
tcp        0      0 localhost:53124         localhost:37017         ESTABLISHED

There is no reference to mongod in the netstat output, but neither for the netstat | grep 27017

Comment by Ramon Fernandez Marina [ 21/May/15 ]

If I understand correctly there's no mongod listening on port 37017. Can you please check with netstat that there's a mongod running on that port?

Comment by Juan Manuel Diego G [ 21/May/15 ]

Just to make sure, I'll include both config files:

Primary process config file

systemLog:
 destination: file
 path: /var/log/mongodb/mongod.log
 logAppend: true
replication:
 replSetName: apData_DEV_01
processManagement:
 pidFilePath: /var/run/mongodb/mongod.pid
 fork: true
net:
 bindIp: 172.31.48.41,127.0.0.1
 port: 27017
 http:
  enabled: false
storage:
 dbPath: /disk1/data
 journal:
  enabled: true
 engine: wiredTiger
security:
 authorization: enabled
 keyFile: /disk1/mongodb-keyfile

Secondary process config file

systemLog:
 destination: file
 path: /var/log/mongodb/mongodRepl02.log
 logAppend: true
replication:
 replSetName: apData_DEV_01
processManagement:
 pidFilePath: /var/run/mongodb/mongodRepl02.pid
 fork: true
net:
 bindIp: 172.31.48.41,127.0.0.1
 port: 37017
 http:
  enabled: false
storage:
 dbPath: /disk1/dataRepl02
 journal:
  enabled: true
 engine: wiredTiger
security:
 authorization: enabled
 keyFile: /disk1/mongodb-keyfile

Generated at Thu Feb 08 03:48:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.