Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-8242

assertion failures (btree) during repair

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.2.0
    • Component/s: Admin
    • Labels:
      None
    • Environment:
      ubuntu on EC2
    • ALL

      Was running a repair with the following command and got this assertion error in the log of 1 out of 3 shards. The other shards had a different assertion error, which I filed a different bug for:

      ubuntu@mongo3:~$ mongod --version
      db version v2.2.0, pdfile version 4.5
      Fri Jan 18 20:56:42 git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207

      sudo -u mongodb mongod --dbpath=/db/mongodb --repair --repairpath=/export/mongodb --nojournal

      Fri Jan 18 03:25:59 [initandlisten] 47127700/861940368 5%
      Fri Jan 18 03:26:09 [initandlisten] 47232800/861940368 5%
      Fri Jan 18 03:26:19 [initandlisten] 47342500/861940368 5%
      Fri Jan 18 03:26:29 [initandlisten] 47451000/861940368 5%
      Fri Jan 18 03:26:39 [initandlisten] 47560200/861940368 5%
      Fri Jan 18 03:26:49 [initandlisten] 47665300/861940368 5%
      Fri Jan 18 03:26:59 [initandlisten] 47774400/861940368 5%
      Fri Jan 18 03:27:09 [initandlisten] 47881800/861940368 5%
      Fri Jan 18 03:27:19 [initandlisten] 47988500/861940368 5%
      Fri Jan 18 03:27:29 [initandlisten] 48097900/861940368 5%
      Fri Jan 18 03:27:39 [initandlisten] 48207400/861940368 5%
      Fri Jan 18 03:27:49 [initandlisten] 48313200/861940368 5%
      Fri Jan 18 03:27:59 [initandlisten] 48422400/861940368 5%
      Fri Jan 18 03:28:00 [initandlisten] pb3 Assertion failure la <= 0xffffff src/mongo/db/btree.h 243
      0xade6e1 0x803dfd 0x683ebb 0xb1e868 0xae7d19 0xae15ed 0xae46e7 0x747ed3 0x7490bd 0xae2c0d 0x737f9c 0x8535bc 0x8560c9 0x856269 0x74037a 0x5596d8 0x55b78d 0x55c6f0 0x55d2dd 0x563dbe
      mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
      mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0x803dfd]
      mongod(ZN5mongo12BucketBasicsINS_12BtreeData_V1EE9_pushBackENS_7DiskLocERKNS_5KeyV1ERKNS_8OrderingES3+0x8ab) [0x683ebb]
      mongod(_ZN5mongo12BtreeBuilderINS_12BtreeData_V1EE6addKeyERNS_7BSONObjENS_7DiskLocE+0x2a8) [0xb1e868]
      mongod(_ZN5mongo24buildBottomUpPhases2And3INS_12BtreeData_V1EEEvbRNS_12IndexDetailsERNS_21BSONObjExternalSorterEbRSt3setINS_7DiskLocESt4lessIS7_ESaIS7_EEPNS_5CurOpEPNS_12SortPhaseOneERNS_19ProgressMeterHolderERNS_5TimerE+0x1f9) [0xae7d19]
      mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0xc9d) [0xae15ed]
      mongod(_ZN5mongo12buildAnIndexESsPNS_16NamespaceDetailsERNS_12IndexDetailsEib+0x217) [0xae46e7]
      mongod(_ZN5mongo16insert_makeIndexEPNS_16NamespaceDetailsERKSsRKNS_7DiskLocE+0x1f3) [0x747ed3]
      mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbPb+0x93d) [0x7490bd]
      mongod(_ZN5mongo17ensureHaveIdIndexEPKc+0x7ad) [0xae2c0d]
      mongod(_ZN5mongo21ensureIdIndexForNewNsEPKc+0x9c) [0x737f9c]
      mongod(_ZN5mongo6Cloner2goEPKcRKNS_12CloneOptionsERSt3setISsSt4lessISsESaISsEERSsPi+0xd1c) [0x8535bc]
      mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbbbbPi+0x139) [0x8560c9]
      mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbbbbPi+0x59) [0x856269]
      mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x52a) [0x74037a]
      mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x68) [0x5596d8]
      mongod() [0x55b78d]
      mongod(_ZN5mongo14_initAndListenEi+0x480) [0x55c6f0]
      mongod(_ZN5mongo13initAndListenEi+0x1d) [0x55d2dd]
      mongod() [0x563dbe]
      Fri Jan 18 03:28:00 [initandlisten] Assertion: 16394:reading doc for external sort failed:errno:2 No such file or directory
      0xade6e1 0x8036eb 0x80386c 0x909535 0x909c10 0xae7bff 0xae15ed 0xae46e7 0x747ed3 0x7490bd 0xae2c0d 0x737f9c 0x8535bc 0x8560c9 0x856269 0x74037a 0x5596d8 0x55b78d 0x55c6f0 0x55d2dd
      mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
      mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0x8036eb]
      mongod() [0x80386c]
      mongod(_ZN5mongo21BSONObjExternalSorter12FileIterator4nextEv+0xc5) [0x909535]
      mongod(_ZN5mongo21BSONObjExternalSorter8Iterator4nextEv+0x220) [0x909c10]
      mongod(_ZN5mongo24buildBottomUpPhases2And3INS_12BtreeData_V1EEEvbRNS_12IndexDetailsERNS_21BSONObjExternalSorterEbRSt3setINS_7DiskLocESt4lessIS7_ESaIS7_EEPNS_5CurOpEPNS_12SortPhaseOneERNS_19ProgressMeterHolderERNS_5TimerE+0xdf) [0xae7bff]
      mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0xc9d) [0xae15ed]
      mongod(_ZN5mongo12buildAnIndexESsPNS_16NamespaceDetailsERNS_12IndexDetailsEib+0x217) [0xae46e7]
      mongod(_ZN5mongo16insert_makeIndexEPNS_16NamespaceDetailsERKSsRKNS_7DiskLocE+0x1f3) [0x747ed3]
      mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbPb+0x93d) [0x7490bd]
      mongod(_ZN5mongo17ensureHaveIdIndexEPKc+0x7ad) [0xae2c0d]
      mongod(_ZN5mongo21ensureIdIndexForNewNsEPKc+0x9c) [0x737f9c]
      mongod(_ZN5mongo6Cloner2goEPKcRKNS_12CloneOptionsERSt3setISsSt4lessISsESaISsEERSsPi+0xd1c) [0x8535bc]
      mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbbbbPi+0x139) [0x8560c9]
      mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbbbbPi+0x59) [0x856269]
      mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x52a) [0x74037a]
      mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x68) [0x5596d8]
      mongod() [0x55b78d]
      mongod(_ZN5mongo14_initAndListenEi+0x480) [0x55c6f0]
      mongod(_ZN5mongo13initAndListenEi+0x1d) [0x55d2dd]
      Fri Jan 18 03:28:08 [initandlisten] exception in initAndListen: 16394 reading doc for external sort failed:errno:2 No such file or directory, terminating
      Fri Jan 18 03:28:08 dbexit:
      Fri Jan 18 03:28:08 [initandlisten] shutdown: going to close listening sockets...
      Fri Jan 18 03:28:08 [initandlisten] shutdown: going to flush diaglog...
      Fri Jan 18 03:28:08 [initandlisten] shutdown: going to close sockets...
      Fri Jan 18 03:28:08 [initandlisten] shutdown: waiting for fs preallocator...
      Fri Jan 18 03:28:08 [initandlisten] shutdown: closing all files...
      Fri Jan 18 03:28:10 [initandlisten] 119/824 14%
      Fri Jan 18 03:28:12 [initandlisten] 402/824 48%
      Fri Jan 18 03:28:14 [initandlisten] 448/824 54%
      Fri Jan 18 03:28:16 [initandlisten] 455/824 55%
      Fri Jan 18 03:28:18 [initandlisten] 464/824 56%
      Fri Jan 18 03:28:20 [initandlisten] 573/824 69%
      Fri Jan 18 03:28:22 [initandlisten] 580/824 70%
      Fri Jan 18 03:28:24 [initandlisten] 605/824 73%
      Fri Jan 18 03:28:26 [initandlisten] 671/824 81%
      Fri Jan 18 03:28:27 [initandlisten] closeAllFiles() finished
      Fri Jan 18 03:28:27 [initandlisten] shutdown: removing fs lock...
      Fri Jan 18 03:28:27 dbexit: really exiting now

        1. mongo1_repair.log
          4 kB
        2. mongo2_repair.log
          4 kB
        3. mongo3_repair.log
          6 kB

            Assignee:
            Unassigned Unassigned
            Reporter:
            zacwitte Zac Witte
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: