[SERVER-38331] checkpoint-server: checkpoint server error: No space left on device Created: 30/Nov/18  Updated: 05/Dec/18  Resolved: 05/Dec/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.11
Fix Version/s: None

Type: Question Priority: Critical - P2
Reporter: Jorge Campos Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: needs_info
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux debian 9 as environment. mongodb version v3.2.11 installed on linux debian..


Participants:

 Description   

Hello,

We have mongodb version v3.2.11 installed on linux debian 9, everyday the mongodb service fails and it stopped. We got the next message in to the log :

2018-11-30T06:50:49.311-0600 E STORAGE [thread1] WiredTiger (28) [1543582249:311004][4637:0x7efc77453700], checkpoint-server: checkpoint server error: No space left on device "

It is stranger because we have space on our particions

S.ficheros Tamaño Usados Disp Uso% Montado en
udev 7,9G 0 7,9G 0% /dev
tmpfs 1,6G 150M 1,5G 10% /run
/dev/mapper/vg0-lv--root 2,0G 389M 1,5G 21% /
/dev/mapper/vg0-lv--usr 5,9G 1,6G 4,0G 29% /usr
tmpfs 7,9G 8,0K 7,9G 1% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 7,9G 0 7,9G 0% /sys/fs/cgroup
/dev/mapper/vg0-lv--tmp 2,0G 1,5M 1,8G 1% /tmp
/dev/mapper/vg0-lv--opt 465M 2,3M 434M 1% /opt
/dev/mapper/vg0-lv--home 2,0G 258M 1,6G 14% /home
/dev/mapper/vg0-lv--var 130G 114G 9,9G 92% /var
/dev/sda1 915M 35M 832M 5% /boot
/dev/mapper/vg0-lv--log 2,0G 411M 1,5G 22% /var/log 
//192.168.2.202/home/backup.db 17T 8,4T 7,9T 52% /mnt/backup.nas 
tmpfs 1,6G 0 1,6G 0% /run/user/1001
 
 
S.ficheros Nodos-i NUsados NLibres NUso% Montado en
udev 2051685 428 2051257 1% /dev
tmpfs 2054669 760 2053909 1% /run
/dev/mapper/vg0-lv--root 124928 8032 116896 7% /
/dev/mapper/vg0-lv--usr 391680 60192 331488 16% /usr
tmpfs 2054669 3 2054666 1% /dev/shm
tmpfs 2054669 4 2054665 1% /run/lock
tmpfs 2054669 15 2054654 1% /sys/fs/cgroup
/dev/mapper/vg0-lv--tmp 124928 15 124913 1% /tmp
/dev/mapper/vg0-lv--opt 124928 11 124917 1% /opt
/dev/mapper/vg0-lv--home 124928 10960 113968 9% /home
/dev/mapper/vg0-lv--var 8464896 25459 8439437 1% /var
/dev/sda1 499712 338 499374 1% /boot
/dev/mapper/vg0-lv--log 124928 256 124672 1% /var/log
//192.168.2.202/home/backup.db 548540416 10962999 537577417 2% /mnt/backup.nas
tmpfs 2054669 11 2054658 1% /run/user/1001
sysadmin@COMPASSCRI-DBS01:~$

How can we solve this issue? Because I need to find the fix about it because our operations are criticals
we appreciate your help about this issue.

 

Regards...

 

 



 Comments   
Comment by Jorge Campos [ 05/Dec/18 ]

Hello Daniel

Thank you very much for your help. we will follow your advice.

Regards...

Comment by Danny Hatcher (Inactive) [ 05/Dec/18 ]

Hello Jorge,

The message you are receiving is related to the available hard drive capacity. In the two outputs of df that you have posted on this case, we can see the usage of /var rise from 92% to 94% so there is clearly some growth in your data. It is possible that whatever processes you are running at the same time is filling up the rest of the drives. We recommend staying outside of the 90s in disk usage as severe problems can occur if the disk does fill up.

Also, I would like to note that your version of MongoDB is fairly old and has reached end-of-life status. Thus, no more bug fixes will be released for 3.2. I recommend upgrading to 3.4, 3.6, or 4.0 at your earliest convenience.

Thank you,

Danny

Comment by Jorge Campos [ 03/Dec/18 ]

Hello Daniel,

Thanks for you help , but I have the last question. The mongodb process run about the 4:00 am with the others process a the same time and this services is stopped by this reason we need to restart de mongodb service and we run de process manually about the 10: 00 am . in this time the process run alone and perfecly. so my questions is.is ther something about the memory or is only hard drive space?

we can't understant why this if we consider the hard drive space. That is strange

 

Regards...

Comment by Danny Hatcher (Inactive) [ 03/Dec/18 ]

Hello Jorge,

Please note that we strongly recommend using a replica set for any Production database.

If an initial sync is not a possibility, we recommend using the compact command as it has the ability to release disk space for Wired Tiger systems. However, it does require some free space in order to run and you may not have enough. Would it be possible to try running compact against your biggest databases? Please note that while the command runs it will block every operation on the specified database until it finishes.

If the compact command is not an option, I recommend you allocate a larger disk to the database as your data has reached the limits of what the current disk can hold.

Thank you,

Danny

Comment by Jorge Campos [ 03/Dec/18 ]

sysadmin@COMPASSCRI-DBS01:/var/lib/mongodb/journal$ ls -lh
total 201M
rw-rr- 1 mongodb mongodb 7,7K dic 3 10:16 WiredTigerLog.0000000032
rw-rr- 1 mongodb mongodb 100M dic 3 09:58 WiredTigerPreplog.0000000001
rw-rr- 1 mongodb mongodb 100M dic 3 09:58 WiredTigerPreplog.0000000002

Comment by Jorge Campos [ 03/Dec/18 ]

Also today we got this message into the log

 

sysadmin@COMPASSCRI-DBS01:/var/log/mongodb$ cat mongodb.log | grep "error"
2018-12-03T09:58:45.170-0600 W FTDC [ftdc] Uncaught exception in 'UnknownError: Caught std::exception of type boost::filesystem::filesystem_error: boost::filesystem::file_size: No such file or directory: "/var/lib/mongodb/diagnostic.data/metrics.interim.temp"' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.

Comment by Jorge Campos [ 03/Dec/18 ]

Hello Daniel,

 

How are you?

That is my mongo configuration.

On another hand we don't use any replica as you can see into the mongod.conf.

 

# mongodb.conf
 
# Where to store the data.
dbpath=/var/lib/mongodb
 
#where to log
logpath=/var/log/mongodb/mongodb.log
 
logappend=true
 
bind_ip = 192.168.9.4
#port = 27017
 
# Enable journaling, http://www.mongodb.org/display/DOCS/Journaling
journal=true
 
# Enables periodic logging of CPU utilization and I/O wait
#cpu = true
 
# Turn on/off security. Off is currently the default
#noauth = true
#auth = true
 
# Verbose logging output.
#verbose = true
 
# Inspect all client data for validity on receipt (useful for
# developing drivers)
#objcheck = true
 
# Enable db quota management
#quota = true
 
# Set oplogging level where n is
# 0=off (default)
# 1=W
# 2=R
# 3=both
# 7=W+some reads
#oplog = 0
 
# Diagnostic/debugging option
#nocursors = true
 
# Ignore query hints
#nohints = true
 
# Disable the HTTP interface (Defaults to localhost:27018).
#nohttpinterface = true
 
# Turns off server-side scripting. This will result in greatly limited
# functionality
#noscripting = true
 
# Turns off table scans. Any query that would do a table scan fails.
#notablescan = true
 
# Disable data file preallocation.
#noprealloc = true
 
# Specify .ns file size for new databases.
 
 
 
# Specify .ns file size for new databases.
# nssize = <size>
 
# Accout token for Mongo monitoring server.
#mms-token = <token>
 
# Server name for Mongo monitoring server.
#mms-name = <server-name>
 
# Ping interval for Mongo monitoring server.
#mms-interval = <seconds>
 
# Replication Options
 
# in replicated mongo databases, specify here whether this is a slave or master
#slave = true
#source = master.example.com
# Slave only: specify a single database to replicate
#only = master.example.com
# or
#master = true
#source = slave.example.com
 
 
# Address of a server to pair with.
#pairwith = <server:port>
# Address of arbiter server.
#arbiter = <server:port>
# Automatically resync if slave data is stale
#autoresync
# Custom size for replication operation log.
#oplogSize = <MB>
# Size limit for in-memory storage of op ids.
#opIdMem = <bytes>
 
# SSL options
# Enable SSL on normal ports
#sslOnNormalPorts = true
# SSL Key file and password
#sslPEMKeyFile = /etc/ssl/mongodb.pem
#sslPEMKeyPassword = pass


 

 

Moreover I show you the path where is locate the mongo

sysadmin@COMPASSCRI-DBS01:/var/lib/mongodb$ ls -ltr
total 435840
-rw-r--r-- 1 mongodb mongodb 21 ago 31 17:10 WiredTiger.lock
-rw-r--r-- 1 mongodb mongodb 46 ago 31 17:10 WiredTiger
-rw-r--r-- 1 mongodb mongodb 95 ago 31 17:10 storage.bson
-rw-r--r-- 1 mongodb mongodb 32768 ago 31 19:02 index-5-4134350373354897081.wt
-rw-r--r-- 1 mongodb mongodb 36864 nov 29 09:54 index-14-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 nov 29 09:54 index-18-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 nov 29 09:54 index-10-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 21897216 dic 3 09:58 index-1-2549256130078277627.wt
-rw-r--r-- 1 mongodb mongodb 422916096 dic 3 09:58 collection-0-2549256130078277627.wt
-rw-r--r-- 1 mongodb mongodb 4096 dic 3 09:58 WiredTigerLAS.wt
drwxr-xr-x 2 mongodb mongodb 4096 dic 3 09:58 journal
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 09:58 _mdb_catalog.wt
-rw-r--r-- 1 mongodb mongodb 32768 dic 3 09:58 collection-4-4134350373354897081.wt
-rw-r--r-- 1 mongodb mongodb 16384 dic 3 09:58 collection-2-4134350373354897081.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 09:58 collection-17-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 09:58 collection-9-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 09:58 collection-13-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 09:58 collection-2-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 09:58 collection-0-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 09:58 collection-5-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 6 dic 3 09:58 mongod.lock
-rw-r--r-- 1 mongodb mongodb 16384 dic 3 09:58 index-3-4134350373354897081.wt
drwxr-xr-x 2 mongodb mongodb 4096 dic 3 09:58 diagnostic.data
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 09:59 index-1-4134350373354897081.wt
-rw-r--r-- 1 mongodb mongodb 40960 dic 3 09:59 collection-0-4134350373354897081.wt
-rw-r--r-- 1 mongodb mongodb 32768 dic 3 10:02 index-6-4134350373354897081.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 10:11 index-1-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 10:11 index-4-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 10:15 index-7-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 10:15 index-8-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 10:15 index-6-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 10:15 index-3-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 69632 dic 3 10:16 index-16-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 77824 dic 3 10:16 index-12-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 163840 dic 3 10:16 collection-11-2723153518991757125.wt
-rw-r--r-- 1 mongodb mongodb 36864 dic 3 10:16 sizeStorer.wt

S.ficheros Tamaño Usados Disp Uso% Montado en 
udev 7,9G 0 7,9G 0% /dev 
tmpfs 1,6G 158M 1,5G 10% /run 
/dev/mapper/vg0-lv--root 2,0G 389M 1,5G 21% / 
/dev/mapper/vg0-lv--usr 5,9G 1,6G 4,0G 29% /usr 
tmpfs 7,9G 8,0K 7,9G 1% /dev/shm 
tmpfs 5,0M 0 5,0M 0% /run/lock 
tmpfs 7,9G 0 7,9G 0% /sys/fs/cgroup 
/dev/mapper/vg0-lv--tmp 2,0G 1,5M 1,8G 1% /tmp 
/dev/mapper/vg0-lv--opt 465M 2,3M 434M 1% /opt 
/dev/mapper/vg0-lv--home 2,0G 258M 1,6G 14% /home 
/dev/mapper/vg0-lv--var 130G 116G 7,8G 94% /var 
/dev/sda1 915M 35M 832M 5% /boot 
/dev/mapper/vg0-lv--log 2,0G 329M 1,6G 18% /var/log 
//192.168.2.202/home/backup.db 17T 8,4T 7,9T 52% /mnt/backup.nas

sysadmin@COMPASSCRI-DBS01:/var/lib/mongodb$ df -i
S.ficheros Nodos-i NUsados NLibres NUso% Montado en
udev 2051685 428 2051257 1% /dev
tmpfs 2054669 757 2053912 1% /run
/dev/mapper/vg0-lv--root 124928 8032 116896 7% /
/dev/mapper/vg0-lv--usr 391680 60192 331488 16% /usr
tmpfs 2054669 3 2054666 1% /dev/shm
tmpfs 2054669 4 2054665 1% /run/lock
tmpfs 2054669 15 2054654 1% /sys/fs/cgroup
/dev/mapper/vg0-lv--tmp 124928 16 124912 1% /tmp
/dev/mapper/vg0-lv--opt 124928 11 124917 1% /opt
/dev/mapper/vg0-lv--home 124928 10960 113968 9% /home
/dev/mapper/vg0-lv--var 8464896 25469 8439427 1% /var
/dev/sda1 499712 338 499374 1% /boot
/dev/mapper/vg0-lv--log 124928 257 124671 1% /var/log
//192.168.2.202/home/backup.db 548540416 11042591 537497825 3% /mnt/backup.nas
tmpfs 2054669 11 2054658 1% /run/user/1001
sysadmin@COMPASSCRI-DBS01:/var/lib/mongodb$

 

Can you help us please ?
Could we do check something according the mongo configuration or to check something in special ?

we will appreciate your help
regards..

 

 

Comment by Danny Hatcher (Inactive) [ 03/Dec/18 ]

Hello Jorge,

Can you confirm where your $dbpath and journal files are located? If they are both on /var, then the drive is full enough that the df output may not be entirely accurate. If you initial sync the node from another member of a replica set, it may free up some space on the disk. Would that be possible to try?

Thank you,

Danny

Generated at Thu Feb 08 04:48:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.