[SERVER-3759] filesystem ops may cause termination when no space left on device Created: 04/Sep/11  Updated: 06/Dec/22  Resolved: 18/Dec/18

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aaron Staple Assignee: Backlog - Storage Execution Team
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-3225 Out of disk space while building new ... Closed
is duplicated by SERVER-8783 Crash when out of space Closed
Related
related to SERVER-3430 Exception while running a command-lin... Closed
related to SERVER-14725 In 2.6, mongod does not resume writin... Closed
related to SERVER-13811 Deal better/Fail more gracefully when... Backlog
related to SERVER-6387 Check before --repair touches data to... Closed
related to SERVER-13224 User defined steps to handle out of d... Closed
related to SERVER-2473 Avoid "no space left" on a shard Closed
is related to SERVER-4038 mongod crashes preallocating next fil... Closed
is related to SERVER-9552 when replica set member has full disk... Backlog
Assigned Teams:
Storage Execution
Operating System: ALL
Participants:

 Description   

eg SERVER-3225

Wed Jun 8 21:23:10 [conn665] CMD: drop gravity_test.artists
Wed Jun 8 21:23:10 [conn665] building new index on

{ _id: 1 }

for gravity_test.users
Wed Jun 8 21:23:10 [conn665] Uncaught std::exception: boost::filesystem::create_directory: No space left on device: "/var/lib/mongodb/_tmp/esort.1307586190.1030033087", terminating
Wed Jun 8 21:23:10 dbexit:

In the above case it looks like the failure occurs when we try to generate an external sort directory in extsort.cpp, but there are other cases that can be problematic. It looks like we are calling dbexit such cases because a boost exception is not an expected mongo exception.

In another example, there may also be an issue when running out of space writing to the external sort files SERVER-3430:

Mon Jul 18 13:42:04 [initandlisten] building new index on

{ status: 1, processed_by: 1, date_queued: 1 }

for queue.metaevent background
979700/10216298 9%
1006500/10216298 9%
1852600/10216298 18%
2370700/10216298 23%
3000000/10216298 29%
4000000/10216298 39%
5000000/10216298 48%
6000000/10216298 58%
7000000/10216298 68%
8000000/10216298 78%
9000000/10216298 88%
10000000/10216298 97%
Mon Jul 18 13:44:31 [initandlisten] external sort used : 11 files in 147 secs
Mon Jul 18 13:44:31 [initandlisten] Assertion: 10446:mmap: can't map area of size 0 file: /data/db/_tmp/esort.1310996524.1314405272//file.7
0x55f5aa 0x5ba9f8 0x7a5eb5 0x7a994e 0x776b40 0x77874d 0x77a0ae 0x77b4cf 0x726458 0x727dcd 0x728e0d 0x78349a 0x8aab98 0x8abad4 0x8acecc 0x8ada88 0x8b36bf 0x7f4c7a53fc1d 0x4e10c9
mongod(_ZN5mongo11msgassertedEiPKc+0x12a) [0x55f5aa]
mongod(_ZN5mongo16MemoryMappedFile3mapEPKcRyi+0x438) [0x5ba9f8]
mongod(_ZN5mongo21BSONObjExternalSorter12FileIteratorC1ESs+0x45) [0x7a5eb5]
mongod(ZN5mongo21BSONObjExternalSorter8IteratorC1EPS0+0x13e) [0x7a994e]
mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0xc70) [0x776b40]
mongod() [0x77874d]
mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0xf4e) [0x77a0ae]
mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x5f) [0x77b4cf]
mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0x3a8) [0x726458]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x10cd) [0x727dcd]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x728e0d]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x46a) [0x78349a]
mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x68) [0x8aab98]
mongod() [0x8abad4]
mongod(_ZN5mongo14_initAndListenEiPKc+0x41c) [0x8acecc]
mongod(_ZN5mongo13initAndListenEiPKc+0x18) [0x8ada88]
mongod(main+0x5acf) [0x8b36bf]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x7f4c7a53fc1d]
mongod(__gxx_personality_v0+0x3a1) [0x4e10c9]
Mon Jul 18 13:44:31 [initandlisten] exception in initAndListen std::exception: mmap: can't map area of size 0 file:


Generated at Thu Feb 08 03:03:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.