[SERVER-42086] Mongod service failed to start Created: 04/Jul/19  Updated: 25/Jul/19  Resolved: 24/Jul/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.12
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Manisha Sharma Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

MongoDB version 3.6
Linux
Recplica set is enabled and this instance is secondary node.


Operating System: ALL
Participants:

 Description   

HI ,

 

I need some help in starting mongod service on one of our servers, the service was running fine before i tried to comment out: wiredTigerDirectoryForIndexes in mongod.conf file.

As of now, the service is stopped and when i try to start it using mongod command it fails with following error:

 

2019-07-03T11:55:31.026+0800 I STORAGE [initandlisten] exception in initAndListen: InvalidOptions: Requested option conflicts with current storage engine option for directoryForIndexes; you requested false but the current server storage is already set to true and cannot be changed, terminating

I have reverted back the changes but it seems not to pick up the chanege. I am not using root acces but i am using sudo option to make any change. Is there any ther option i can try before running mongo repair as this is a production machine.



 Comments   
Comment by Manisha Sharma [ 25/Jul/19 ]

Please fell free to close the ticket. Meanwhile i have provided permission again. mongod service still failed. I will see other options too.

Comment by Danny Hatcher (Inactive) [ 24/Jul/19 ]

As the most recent error says

2019-07-11T13:17:25.364+0800 I STORAGE [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/db, terminating

You need to have permissions to execute on the directory if you wish to use it as your data path.

As the investigation thus far has not revealed a bug, I am going to close this ticket. If you need further assistance troubleshooting, I encourage you to ask our community by posting on the mongodb-user group or on Stack Overflow with the mongodb tag.

Comment by Manisha Sharma [ 24/Jul/19 ]

I have checked /data/db path and it is empty with no data or file in it. So which means i dont have to remove anything from my side. however restart is still not working.

Comment by Manisha Sharma [ 24/Jul/19 ]

Logfile info:

2019-07-03T12:27:51.184+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 172.22.21.184:27017
2019-07-03T12:27:52.191+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 172.22.21.184:27017 - HostUnreachable: No route to host
2019-07-03T12:27:52.191+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 172.22.21.184:27017 due to failed operation on a connection
2019-07-03T12:27:52.191+0800 I REPL_HB [replexec-2] Error in heartbeat (requestId: 792) to 172.22.21.184:27017, response status: HostUnreachable: No route to host
2019-07-03T12:27:52.192+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 172.22.21.184:27017
2019-07-03T12:27:55.200+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 172.22.21.184:27017 - HostUnreachable: No route to host
2019-07-03T12:27:55.200+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 172.22.21.184:27017 due to failed operation on a connection
2019-07-03T12:27:55.200+0800 I REPL_HB [replexec-1] Error in heartbeat (requestId: 794) to 172.22.21.184:27017, response status: HostUnreachable: No route to host
2019-07-03T12:27:55.200+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 172.22.21.184:27017
2019-07-03T12:27:58.205+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 172.22.21.184:27017 - HostUnreachable: No route to host
2019-07-03T12:27:58.205+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 172.22.21.184:27017 due to failed operation on a connection
2019-07-03T12:27:58.205+0800 I REPL_HB [replexec-1] Error in heartbeat (requestId: 800) to 172.22.21.184:27017, response status: HostUnreachable: No route to host
2019-07-03T12:27:59.787+0800 I CONTROL [signalProcessingThread] got signal 2 (Interrupt), will terminate after current cmd ends
2019-07-03T12:27:59.787+0800 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets...
2019-07-03T12:27:59.787+0800 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2019-07-03T12:27:59.787+0800 I REPL [signalProcessingThread] shutdown: removing all drop-pending collections...
2019-07-03T12:27:59.788+0800 I REPL [signalProcessingThread] shutdown: removing checkpointTimestamp collection...
2019-07-03T12:27:59.788+0800 I REPL [signalProcessingThread] shutting down replication subsystems
2019-07-03T12:27:59.788+0800 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
2019-07-03T12:27:59.790+0800 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2019-07-03T12:27:59.839+0800 I STORAGE [signalProcessingThread] WiredTiger message [1562128079:839131][10715:0x7fd610284700], txn-recover: Set global recovery timestamp: 0
2019-07-03T12:27:59.894+0800 I STORAGE [signalProcessingThread] shutdown: removing fs lock...
2019-07-03T12:27:59.894+0800 I CONTROL [signalProcessingThread] now exiting
2019-07-03T12:27:59.894+0800 I CONTROL [signalProcessingThread] shutting down with code:0
2019-07-11T13:17:25.325+0800 I CONTROL [main] ***** SERVER RESTARTED *****
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] MongoDB starting : pid=12606 port=27017 dbpath=/data/db 64-bit host=hkfpmatd02
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] db version v3.6.10
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] git version: 3e3ab85bfb98875af3bc6e74eeb945b0719f69c8
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] allocator: tcmalloc
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] modules: enterprise
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] build environment:
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] distmod: rhel70
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] distarch: x86_64
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] target_arch: x86_64
2019-07-11T13:17:25.351+0800 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

{ bindIp: "172.22.21.185,127.0.0.1", port: 27017 }

, processManagement:

{ pidFilePath: "/var/run/mongodb/mongod.pid" }

, replication:

{ replSetName: "MtxEventReplSet" }

, security:

{ authorization: "enabled", keyFile: "/srv/mongodb/mongodb-keyfile" }

, storage: { dbPath: "/data/db", engine: "wiredTiger", journal:

{ enabled: false }

, wiredTiger: { engineConfig:

{ directoryForIndexes: true }

} }, systemLog:

{ destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" }

}
2019-07-11T13:17:25.364+0800 I STORAGE [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/db, terminating
2019-07-11T13:17:25.364+0800 I CONTROL [initandlisten] now exiting
2019-07-11T13:17:25.364+0800 I CONTROL [initandlisten] shutting down with code:100

Comment by Manisha Sharma [ 24/Jul/19 ]

Please find mongod configuration file:

systemLog:
destination: file
path: /var/log/mongodb/mongod.log
logAppend: true

processManagement:
#fork: true
pidFilePath: /var/run/mongodb/mongod.pid

net:
bindIp: 172.22.21.185,127.0.0.1
port: 27017

storage:
dbPath: /data/db
journal.enabled: false
engine: wiredTiger
wiredTiger:
engineConfig:
directoryForIndexes: true

security:
authorization: enabled
keyFile: /srv/mongodb/mongodb-keyfile

replication:
replSetName: MtxEventReplSet

#sharding:

  1. clusterRole: shardsvr

I will be sharing logfile information in next attachment.

Comment by Danny Hatcher (Inactive) [ 17/Jul/19 ]

As I mentioned in my previous comment, if you are unable to use the database with the original setting please provide the config used to start the server as well as the full mongod log.

Comment by Manisha Sharma [ 17/Jul/19 ]

HI Daniel,

When you say "Remove all the data un the existing db path ", it means to delete the user created databases as well. I can not change the dbpath directory since the prod is already running on it, I have reset the setting if -directoryindex : true but it has not again picked. I want to know, what could be the reason and how can i commit this setting back in the configuration file without deleting db path.

Comment by Danny Hatcher (Inactive) [ 08/Jul/19 ]

As stated in our documentation, you cannot change the --directoryperdb setting on an existing data directory. If you wish to change the setting, you must change the dbpath or remove all the data in the existing dbpath and perform an initial sync.

You should be able to set the setting back to the original value and continue using the database. If you are unable to do so, please provide the config used to start the server as well as the full mongod log.

Comment by Manisha Sharma [ 05/Jul/19 ]

HI , Is there any update on this issue?

Generated at Thu Feb 08 04:59:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.