[SERVER-42208] Permission Denied Error, cannot start DB Created: 12/Jul/19  Updated: 01/Oct/19  Resolved: 01/Oct/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Minor - P4
Reporter: Tobias Eichinger Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Machine: NVIDIA DGX-1
Kernel: 4.15.0-54-generic
OS: Ubuntu 18.04.2 LTS


Participants:

 Description   

Dear MongoDB community,

 

I have the following error messages after running my usual startup command:

 

numactl --interleave=all mongod

###############################

2019-07-12T17:15:42.293+0200 I CONTROL [initandlisten] MongoDB starting : pid=12762 port=27017 dbpath=/data/db 64-bit host=dgx-server
2019-07-12T17:15:42.293+0200 I CONTROL [initandlisten] db version v3.6.3
2019-07-12T17:15:42.293+0200 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2019-07-12T17:15:42.293+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
2019-07-12T17:15:42.293+0200 I CONTROL [initandlisten] allocator: tcmalloc
2019-07-12T17:15:42.293+0200 I CONTROL [initandlisten] modules: none
2019-07-12T17:15:42.293+0200 I CONTROL [initandlisten] build environment:
2019-07-12T17:15:42.293+0200 I CONTROL [initandlisten] distarch: x86_64
2019-07-12T17:15:42.293+0200 I CONTROL [initandlisten] target_arch: x86_64
2019-07-12T17:15:42.293+0200 I CONTROL [initandlisten] options: {}
2019-07-12T17:15:42.293+0200 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-07-12T17:15:42.293+0200 I STORAGE [initandlisten]
2019-07-12T17:15:42.293+0200 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-07-12T17:15:42.293+0200 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-07-12T17:15:42.293+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=257429M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-07-12T17:15:42.809+0200 E STORAGE [initandlisten] WiredTiger error (13) [1562944542:809396][12762:0x7f4fd45dc0c0], file:WiredTiger.wt, connection: /data/db/WiredTiger.turtle: handle-open: open: Permission denied
2019-07-12T17:15:42.810+0200 E - [initandlisten] Assertion: 28595:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 413
2019-07-12T17:15:42.810+0200 I STORAGE [initandlisten] exception in initAndListen: Location28595: 13: Permission denied, terminating
2019-07-12T17:15:42.810+0200 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2019-07-12T17:15:42.810+0200 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2019-07-12T17:15:42.810+0200 I CONTROL [initandlisten] now exiting
2019-07-12T17:15:42.810+0200 I CONTROL [initandlisten] shutting down with code:100

 

Since this is a Permission Denied error, I tried sudoing

sudo numactl --interleave=all mongod

#############################################

 

2019-07-12T17:23:23.845+0200 I CONTROL [initandlisten] MongoDB starting : pid=12772 port=27017 dbpath=/data/db 64-bit host=dgx-server
2019-07-12T17:23:23.845+0200 I CONTROL [initandlisten] db version v3.6.3
2019-07-12T17:23:23.845+0200 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2019-07-12T17:23:23.845+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
2019-07-12T17:23:23.845+0200 I CONTROL [initandlisten] allocator: tcmalloc
2019-07-12T17:23:23.845+0200 I CONTROL [initandlisten] modules: none
2019-07-12T17:23:23.845+0200 I CONTROL [initandlisten] build environment:
2019-07-12T17:23:23.845+0200 I CONTROL [initandlisten] distarch: x86_64
2019-07-12T17:23:23.845+0200 I CONTROL [initandlisten] target_arch: x86_64
2019-07-12T17:23:23.845+0200 I CONTROL [initandlisten] options: {}
2019-07-12T17:23:23.845+0200 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-07-12T17:23:23.845+0200 I STORAGE [initandlisten]
2019-07-12T17:23:23.845+0200 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-07-12T17:23:23.845+0200 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-07-12T17:23:23.845+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=257429M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-07-12T17:23:24.363+0200 E STORAGE [initandlisten] WiredTiger error (-31802) [1562945004:363175][12772:0x7fd8389b30c0], txn-recover: unsupported WiredTiger file version: this build only supports versions up to 2, and the file is version 3: WT_ERROR: non-specific WiredTiger error
2019-07-12T17:23:24.363+0200 E STORAGE [initandlisten] WiredTiger error (0) [1562945004:363199][12772:0x7fd8389b30c0], txn-recover: WiredTiger is unable to read the recovery log.
2019-07-12T17:23:24.363+0200 E STORAGE [initandlisten] WiredTiger error (0) [1562945004:363206][12772:0x7fd8389b30c0], txn-recover: This may be due to the log files being encrypted, being from an older version or due to corruption on disk
2019-07-12T17:23:24.363+0200 E STORAGE [initandlisten] WiredTiger error (0) [1562945004:363211][12772:0x7fd8389b30c0], txn-recover: You should confirm that you have opened the database with the correct options including all encryption and compression options
2019-07-12T17:23:24.363+0200 E STORAGE [initandlisten] WiredTiger error (-31802) [1562945004:363218][12772:0x7fd8389b30c0], txn-recover: Recovery failed: WT_ERROR: non-specific WiredTiger error
2019-07-12T17:23:24.364+0200 E - [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 413
2019-07-12T17:23:24.364+0200 I STORAGE [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2019-07-12T17:23:24.364+0200 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2019-07-12T17:23:24.364+0200 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2019-07-12T17:23:24.364+0200 I CONTROL [initandlisten] now exiting
2019-07-12T17:23:24.364+0200 I CONTROL [initandlisten] shutting down with code:100

 

I am sorry but I have no clue what causes this issue since it has been starting up perfectly fine in the past. Perhaps there was a crash on the server that I am co-working with others. Do you know what I can do to resolve this?

Kind regards

 



 Comments   
Comment by Danny Hatcher (Inactive) [ 04/Sep/19 ]

Have you had a chance to follow my instructions?

Comment by Danny Hatcher (Inactive) [ 06/Aug/19 ]

In general, I recommend using our official documentation located at docs.mongodb.com but the site you linked does appear to be accurate for the download step.

The error you see is related to dpkg not liking that another version exists. You should be able to run

sudo dpkg -P mongodb-org

then try to run the install again.

The admin command you mentioned uses the mongo shell that comes with the binaries you download. After downloading the package, use the mongo command. Then enter the following into the new prompt:

db.adminCommand( { setFeatureCompatibilityVersion: "4.0" } )

Comment by Tobias Eichinger [ 06/Aug/19 ]

Hey, sorry for the late response, summer break...

I tried installing (additionally to the 3.6.3 binaries) the 4.0.1 binaries for testing. I followed the instructions in

https://tecadmin.net/install-mongodb-on-ubuntu/

until I reached another dead end. When I try to install the binaries via

sudo apt install mongodb-org=4.0.1 mongodb-org-server=4.0.1 mongodb-org-shell=4.0.1 mongodb-org-mongos=4.0.1 mongodb-org-tools=4.0.1

 

Hit:1 https://download.docker.com/linux/ubuntu bionic InRelease
Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease
Ign:4 https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 InRelease
libcurl4
The following packages will be REMOVED:
libcurl3 mongodb mongodb-clients mongodb-server
The following NEW packages will be installed:
libcurl4 mongodb-org mongodb-org-mongos mongodb-org-server mongodb-org-shell mongodb-org-tools
0 upgraded, 6 newly installed, 4 to remove and 54 not upgraded.
Need to get 55,8 MB of archives.
After this operation, 168 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://security.ubuntu.com/ubuntu bionic-security/main amd64 libcurl4 amd64 7.58.0-2ubuntu3.7 [214 kB]
Get:2 https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0/multiverse amd64 mongodb-org-shell amd64 4.0.1 [9.721 kB]
Get:3 https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0/multiverse amd64 mongodb-org-server amd64 4.0.1 [15,5 MB]
Get:4 https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0/multiverse amd64 mongodb-org-mongos amd64 4.0.1 [8.676 kB]
Get:5 https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0/multiverse amd64 mongodb-org-tools amd64 4.0.1 [21,6 MB]
Get:6 https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0/multiverse amd64 mongodb-org amd64 4.0.1 [3.528 B]
Fetched 55,8 MB in 2s (36,2 MB/s)
(Reading database ... 240727 files and directories currently installed.)
Removing libcurl3:amd64 (7.58.0-2ubuntu2) ...
Removing mongodb (1:3.6.3-0ubuntu1.1) ...
Removing mongodb-server (1:3.6.3-0ubuntu1.1) ...
Removing mongodb-clients (1:3.6.3-0ubuntu1.1) ...
Selecting previously unselected package libcurl4:amd64.
(Reading database ... 240702 files and directories currently installed.)
Preparing to unpack .../0-libcurl4_7.58.0-2ubuntu3.7_amd64.deb ...
Unpacking libcurl4:amd64 (7.58.0-2ubuntu3.7) ...
Selecting previously unselected package mongodb-org-shell.
Preparing to unpack .../1-mongodb-org-shell_4.0.1_amd64.deb ...
Unpacking mongodb-org-shell (4.0.1) ...
Selecting previously unselected package mongodb-org-server.
Preparing to unpack .../2-mongodb-org-server_4.0.1_amd64.deb ...
Unpacking mongodb-org-server (4.0.1) ...
dpkg: error processing archive /tmp/apt-dpkg-install-8CHt8o/2-mongodb-org-server_4.0.1_amd64.deb (--unpack):
trying to overwrite '/usr/bin/mongod', which is also in package mongodb-server-core 1:3.6.3-0ubuntu1.1
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Selecting previously unselected package mongodb-org-mongos.
Preparing to unpack .../3-mongodb-org-mongos_4.0.1_amd64.deb ...
Unpacking mongodb-org-mongos (4.0.1) ...
dpkg: error processing archive /tmp/apt-dpkg-install-8CHt8o/3-mongodb-org-mongos_4.0.1_amd64.deb (--unpack):
trying to overwrite '/usr/bin/mongos', which is also in package mongodb-server-core 1:3.6.3-0ubuntu1.1
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Selecting previously unselected package mongodb-org-tools.
Preparing to unpack .../4-mongodb-org-tools_4.0.1_amd64.deb ...
Unpacking mongodb-org-tools (4.0.1) ...
Selecting previously unselected package mongodb-org.
Preparing to unpack .../5-mongodb-org_4.0.1_amd64.deb ...
Unpacking mongodb-org (4.0.1) ...
Errors were encountered while processing:
/tmp/apt-dpkg-install-8CHt8o/2-mongodb-org-server_4.0.1_amd64.deb
/tmp/apt-dpkg-install-8CHt8o/3-mongodb-org-mongos_4.0.1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

It seems as if it finds the download locations, downloads them, yet cannot process the archives via dpkg... It also seems that upgrading my verison 3.6.3 to 4.0.1 is an option, yet I cannot follow the instructions in your docs.. I do not understand where to enter for example:

db.adminCommand( { setFeatureCompatibility : Version} )

It does not seem to be my bash shell. Perhaps I can also add it to the config file? [I wanted to follow the instructions here:https://docs.mongodb.com/manual/release-notes/4.0-upgrade-standalone/ ]

Sorry again for my ignorance on the matter.

 

 

Comment by Danny Hatcher (Inactive) [ 30/Jul/19 ]

Have you had a chance to try the 4.0 binaries?

Comment by Danny Hatcher (Inactive) [ 19/Jul/19 ]

I'm not sure what happened but the only situation this error should occur is when a database has Feature Compatibility Version 4.0 and that can only happen if the node was started as 4.0.x at one point. Regardless of how it occurred, could you please try install 4.0.10 binaries and start the node with them? That will at least confirm the problem.

Comment by Tobias Eichinger [ 16/Jul/19 ]

I started another method, since I was not sure whether to use numactl or sysctl. since I am using a NUMA machine I went back to numactl, yet I did not find any mongod version 4.x, the version that I have installed (/usr/bin/mongod) is version 3.6.3... I do not grasp this problem, where could the version 4.x binary be? I have not installed another version. I have asked my colleagues, they did not touch it either.

/usr/bin/mongod --version

::

db version v3.6.3
git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
allocator: tcmalloc
modules: none
build environment:
distarch: x86_64
target_arch: x86_64

Comment by Danny Hatcher (Inactive) [ 15/Jul/19 ]

You need to set the parameter on a currently running version of the database. In the first example you provided you were starting mongod with numactl but the most recent example you are using systemctl. Which do you use to normally start the server? You should use the same command you normally do but with the 4.0.10 binaries instead of 3.6.3.

Comment by Tobias Eichinger [ 15/Jul/19 ]

Thank you Mr. Hatcher for your timely reply.

I did the following

  1. 1. Start the server with 4.0 binaries with feature compatibility version 4.0
    sudo systemctl start mongodb
  2. this produces an error stating that
  3. :: Failed to start mongodb.service: Unit mongodb.service is masked.
  4. when a file is masked, it is hard disabled, all symlinks point to dev/null, and any activity even manually is discabled
  5. we need to unmask first https://askubuntu.com/questions/710420/why-are-some-systemd-services-in-the-masked-state
  6. 1.a Unmask mongodb.service
    teichinger@dgx-server:~$ sudo systemctl unmask mongodb
  7. :: Removed /etc/systemd/system/mongodb.service.
    sudo systemctl start mongodb
  8. :: Job for mongodb.service failed because the control process exited with error code.
    See "systemctl status mongodb.service" and "journalctl -xe" for details.
  9. 1.b check the status of mongodb.service
    teichinger@dgx-server:~$ systemctl status mongodb.service
    ● mongodb.service - LSB: An object/document-oriented database
    Loaded: loaded (/etc/init.d/mongodb; generated)
    Active: failed (Result: exit-code) since Mon 2019-07-15 11:18:10 CEST; 11s ago
    Docs: man:systemd-sysv-generator(8)
    Process: 15947 ExecStart=/etc/init.d/mongodb start (code=exited, status=1/FAILURE)

I am puzzled what the help page for the compatibility version tells me. I tried having a look at the config option, but I did not find anything there.

Could you please provide a line of code that starts the 4.0 binaries with feature compatibility version 4.0? I am sure that there is a command line option, but I did not find anything on the docu page..

mongod --setParameter featureCompatibilityVersion=4.0 [did not work]

 

 

 

Comment by Danny Hatcher (Inactive) [ 12/Jul/19 ]

There are two different errors here; a permission denied error and an indication that you are using a different version of MongoDB than the data expects.

2019-07-12T17:23:24.363+0200 E STORAGE [initandlisten] WiredTiger error (-31802) [1562945004:363175][12772:0x7fd8389b30c0], txn-recover: unsupported WiredTiger file version: this build only supports versions up to 2, and the file is version 3: WT_ERROR: non-specific WiredTiger error

The above error will only appear if the server was once started with 4.0 binaries and has the feature compatibility version set to 4.0. If you start up the server using 4.0 binaries, you should be able to change the feature compatibility version to 3.6, cleanly shutdown, then successfully start up as a 3.6 server.

Could you please do the above and see if it resolves both issues or at least the second?

Generated at Thu Feb 08 04:59:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.