Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3759

filesystem ops may cause termination when no space left on device

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Storage
    • Labels:
      None
    • Storage Execution
    • ALL

      eg SERVER-3225

      Wed Jun 8 21:23:10 [conn665] CMD: drop gravity_test.artists
      Wed Jun 8 21:23:10 [conn665] building new index on

      { _id: 1 }

      for gravity_test.users
      Wed Jun 8 21:23:10 [conn665] Uncaught std::exception: boost::filesystem::create_directory: No space left on device: "/var/lib/mongodb/_tmp/esort.1307586190.1030033087", terminating
      Wed Jun 8 21:23:10 dbexit:

      In the above case it looks like the failure occurs when we try to generate an external sort directory in extsort.cpp, but there are other cases that can be problematic. It looks like we are calling dbexit such cases because a boost exception is not an expected mongo exception.

      In another example, there may also be an issue when running out of space writing to the external sort files SERVER-3430:

      Mon Jul 18 13:42:04 [initandlisten] building new index on

      { status: 1, processed_by: 1, date_queued: 1 }

      for queue.metaevent background
      979700/10216298 9%
      1006500/10216298 9%
      1852600/10216298 18%
      2370700/10216298 23%
      3000000/10216298 29%
      4000000/10216298 39%
      5000000/10216298 48%
      6000000/10216298 58%
      7000000/10216298 68%
      8000000/10216298 78%
      9000000/10216298 88%
      10000000/10216298 97%
      Mon Jul 18 13:44:31 [initandlisten] external sort used : 11 files in 147 secs
      Mon Jul 18 13:44:31 [initandlisten] Assertion: 10446:mmap: can't map area of size 0 file: /data/db/_tmp/esort.1310996524.1314405272//file.7
      0x55f5aa 0x5ba9f8 0x7a5eb5 0x7a994e 0x776b40 0x77874d 0x77a0ae 0x77b4cf 0x726458 0x727dcd 0x728e0d 0x78349a 0x8aab98 0x8abad4 0x8acecc 0x8ada88 0x8b36bf 0x7f4c7a53fc1d 0x4e10c9
      mongod(_ZN5mongo11msgassertedEiPKc+0x12a) [0x55f5aa]
      mongod(_ZN5mongo16MemoryMappedFile3mapEPKcRyi+0x438) [0x5ba9f8]
      mongod(_ZN5mongo21BSONObjExternalSorter12FileIteratorC1ESs+0x45) [0x7a5eb5]
      mongod(ZN5mongo21BSONObjExternalSorter8IteratorC1EPS0+0x13e) [0x7a994e]
      mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0xc70) [0x776b40]
      mongod() [0x77874d]
      mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0xf4e) [0x77a0ae]
      mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x5f) [0x77b4cf]
      mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0x3a8) [0x726458]
      mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x10cd) [0x727dcd]
      mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3d) [0x728e0d]
      mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x46a) [0x78349a]
      mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x68) [0x8aab98]
      mongod() [0x8abad4]
      mongod(_ZN5mongo14_initAndListenEiPKc+0x41c) [0x8acecc]
      mongod(_ZN5mongo13initAndListenEiPKc+0x18) [0x8ada88]
      mongod(main+0x5acf) [0x8b36bf]
      /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f4c7a53fc1d]
      mongod(__gxx_personality_v0+0x3a1) [0x4e10c9]
      Mon Jul 18 13:44:31 [initandlisten] exception in initAndListen std::exception: mmap: can't map area of size 0 file:

            Assignee:
            backlog-server-execution [DO NOT USE] Backlog - Storage Execution Team
            Reporter:
            aaron Aaron Staple
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: