[SERVER-36236] "Unable to read the storage engine metadata file" error after upgrading Mongo binaries Created: 23/Jul/18  Updated: 04/Sep/18  Resolved: 30/Jul/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tedi Roca Assignee: Nick Brewer
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Initial setup - works OK

-------------------------------

  • 3.2 Mongo binaries deployed manually.
  • Mongo server is started via a service by executing:  "sleep 60 && /sbin/runuser mongodb -s /bin/bash -c "/path/to/mongo/binaries/mongod --config /path/to/mongo/conf/mongod.conf"

 

Attempt to upgrade Mongo binaries - it does not work

------------------------------------------------------------------------

  • Mongo binaries are replaced with release 3.4.
  • The previous service nor its permissions are not modified.
  • Mongo is started successfully when calling the binary directly (without using the service) by running "/path/to/mongo/binaries/mongod --config /path/to/mongo/conf/mongod.conf"
  • When starting Mongo via the service, the following error appears:

2018-07-23T13:02:56.017+0000 E STORAGE [initandlisten] Unable to read the storage engine metadata file: FileStreamFailed: Unable to read BSON data from /mongodb/storage.bson
2018-07-23T13:02:56.017+0000 I - [initandlisten] Fatal Assertion 28661 at src/mongo/db/storage/storage_engine_metadata.cpp 76
2018-07-23T13:02:56.017+0000 I - [initandlisten]***aborting after fassert() failure
2018-07-23T13:02:56.023+0000 F - [initandlisten] Got signal: 6 (Aborted).0x2b4a90866311 0x2b4a90865529 0x2b4a90865a0d 0x2b4a932695e0 0x2b4a934ab1f7 0x2b4a934ac8e8 0x2b4a8fb00f3f 0x2b4a9053adb0 0x2b4a9053af38

{{ 0x2b4a9043f271 0x2b4a8faed1de 0x2b4a8fb0cecb 0x2b4a93497c05 0x2b4a8fb6c2df}}

 

Permissions of the storage.bson file:

mongodb@c675kgrlyncs:/appserver/mongo> ls -la /mongodb/storage.bson
rw-r---- 1 mongodb mongodb 95 Aug 2 2016 /mongodb/storage.bson

Participants:

 Description   

The error appears after replacing the Mongo binaries from 3.2 with those from release 3.4 and the server is started again via a service. The server starts successfully when calling directly the binaries (without a service).



 Comments   
Comment by Nick Brewer [ 30/Jul/18 ]

tedi That's certainly a possibility. I'm going to close this ticket for now - but feel free to report back with your findings here, and we can re-open if there's any underlying mongod issues.

-Nick

Comment by Tedi Roca [ 30/Jul/18 ]

I've tried with these binaries but it's still showing the same error. We're investigating whether the issue could be related to the encryption software that is running on these instances.

Comment by Nick Brewer [ 24/Jul/18 ]

tedi I'm still having trouble recreating this in my duplicated environment - I suggest trying again with the 3.4 binaries available here, and including the ls -ltrha output of the updated bin directory if you run into this again.

Thanks,
Nick

Comment by Tedi Roca [ 24/Jul/18 ]

I've just uploaded the additional information to the secure portal.

Thanks,

Tedi 

Comment by Nick Brewer [ 23/Jul/18 ]

tedi

Thanks. I reproduced a basic version of your setup, but I wasn't able to recreate the error you're seeing upgrading from 3.2 to 3.4. A few more things I'd like to confirm:

  • The output of ls -ltrha for your MongoDB binary directory.
  • The output of systemctl status for your mongod service, after it fails, as well as the output of journalctl -xe.
  • Whether or not SELinux is enabled.

Thanks,
-Nick

Comment by Tedi Roca [ 23/Jul/18 ]

Hi Nick,

I've uploaded all the details to your secure portal. Please let me know if you need further information.

Thanks,

Tedi

Comment by Nick Brewer [ 23/Jul/18 ]

tedi

Thanks for your report. I'd like to get some additional information:

  • The full service file that is being used to load mongod.
  • The specific platform (distribution, version) on which you're running.
  • The contents of your mongod.conf file. You can feel free to redact any sensitive information, as long as the underlying configuration options are intact.
  • The output of ls -ltrha run within the mongod $dbpath.

Since this is a public ticket, you can upload this information to our secure portal if you'd prefer - information provided there is only available to MongoDB employees, and is automatically removed after a period of time.

-Nick

Generated at Thu Feb 08 04:42:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.