[SERVER-45604] Mongo DB invariant failure Created: 16/Jan/20  Updated: 27/Jan/20  Resolved: 27/Jan/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Hemanth Vemula Assignee: Carl Champain (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-45787 Mongo DB invariant failure Closed
Operating System: ALL
Participants:

 Description   

Hi All,

I have installed linux VM in azure and attached a external disk(1024 GB) to it. when I tried to change the db path from /var/lib/mongo to /newdisk , I am not able to start mongo service. 

Could you please help me with this.

 

 

Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
[azureuser@campuseatingaiVmIisD01 /]$ systemctl status mongod.service
● mongod.service - MongoDB Database Server
 Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
 Active: failed (Result: exit-code) since Thu 2020-01-16 15:17:24 UTC; 13s ago
 Docs: https://docs.mongodb.org/manual
 Process: 46976 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=1/FAILURE)
 Process: 46974 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
 Process: 46972 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
 Process: 46971 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
 Main PID: 46311 (code=exited, status=0/SUCCESS)



 Comments   
Comment by Carl Champain (Inactive) [ 27/Jan/20 ]

vemulahemanth333@gmail.com,

We are glad to know that it's working! 
Closing this ticket.
 

Comment by Hemanth Vemula [ 27/Jan/20 ]

it worked now . thanks a lot 

 

Comment by Carl Champain (Inactive) [ 24/Jan/20 ]

vemulahemanth333@gmail.com,

Can you please:

  1. Make mongod the user (instead of testuser) for aramark and data.
  2. Give us the output for ls -la /aramark/data/.
  3. Invoke mongod --dbpath /aramark/data/, and let us know if it works. 

Thanks,
Carl

Comment by Hemanth Vemula [ 23/Jan/20 ]

anything else do i need to look at? 

 

Comment by Hemanth Vemula [ 22/Jan/20 ]

 

drwxrwxrwx. 5 testuser mongod 4096 Jan 16 22:40 aramark

drwxrwxrwx. 4 testuser mongod 4096 Jan 19 17:06 data

 

 

both the folders have same permissions

Comment by Danny Hatcher (Inactive) [ 22/Jan/20 ]

What about the permissions on /aramark/data specifically?

Comment by Hemanth Vemula [ 22/Jan/20 ]

2020-01-22T17:06:10.689+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

{ bindIp: "127.0.0.1", port: 27017 }

, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid", timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/aramark/data", journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2020-01-22T17:06:10.689+0000 I STORAGE [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /aramark/data, terminating
2020-01-22T17:06:10.689+0000 F - [initandlisten] Invariant failure globalStorageEngine src/mongo/db/service_context_d.cpp 272

 

Comment by Hemanth Vemula [ 22/Jan/20 ]

I am still getting the same error

Comment by Hemanth Vemula [ 22/Jan/20 ]

drwxrwxrwx. 5 mongod mongod 4096 Jan 16 22:40 aramark 

Comment by Danny Hatcher (Inactive) [ 22/Jan/20 ]

Can you please provide the logs from after you provided the correct file permissions?

Comment by Hemanth Vemula [ 22/Jan/20 ]

Hi Carl, 

I would appreciate your help in the above as soon as possible. 

Please let me know if you need anything else from my end.

Comment by Hemanth Vemula [ 19/Jan/20 ]

I have tried giving full permissions to that mount point, still it is failing, 

Could you please help me with this issue .

Thanks 

Hemanth V

Comment by Hemanth Vemula [ 19/Jan/20 ]

Hi Carl,

 

Please find below error from mongodb.log file .

 

 

2020-01-19T16:56:33.996+0000 I CONTROL [initandlisten] MongoDB starting : pid=63348 port=27017 dbpath=/aramark/data 64-bit host=campuseatingaiVmIisD01
2020-01-19T16:56:33.996+0000 I CONTROL [initandlisten] db version v3.6.16
2020-01-19T16:56:33.996+0000 I CONTROL [initandlisten] git version: 204ab367a130a4fd2db1c54b02cd6a86e4e07f56
2020-01-19T16:56:33.996+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2020-01-19T16:56:33.996+0000 I CONTROL [initandlisten] allocator: tcmalloc
2020-01-19T16:56:33.996+0000 I CONTROL [initandlisten] modules: none
2020-01-19T16:56:33.996+0000 I CONTROL [initandlisten] build environment:
2020-01-19T16:56:33.996+0000 I CONTROL [initandlisten] distmod: rhel70
2020-01-19T16:56:33.996+0000 I CONTROL [initandlisten] distarch: x86_64
2020-01-19T16:56:33.996+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-01-19T16:56:33.997+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

{ bindIp: "127.0.0.1", port: 27017 }

, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid", timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/aramark/data", journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2020-01-19T16:56:33.997+0000 I STORAGE [initandlisten] exception in initAndListen: Location28596: Unable to determine status of lock file in the data directory /aramark/data: boost::filesystem::status: Permission denied: "/aramark/data/mongod.lock", terminating
2020-01-19T16:56:33.997+0000 F - [initandlisten] Invariant failure globalStorageEngine src/mongo/db/service_context_d.cpp 272
*2020-01-19T16:56:33.997+0000 F - [initan*dlisten]

***aborting after invariant() failure

Comment by Carl Champain (Inactive) [ 17/Jan/20 ]

Hi vemulahemanth333@gmail.com,

Thanks for the report.
To help us investigate this issue, can you please share the mongod.log file for this node?

Kind regards,
Carl

Generated at Thu Feb 08 05:09:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.