Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16236

failed to create WiredTiger bulk cursor: Device or resource busy

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • 2.8.0-rc2
    • Affects Version/s: None
    • Component/s: Storage
    • None
    • Fully Compatible
    • ALL

      2014-11-19T16:21:58.488+0000 I CONTROL  [conn72] pid=20764 port=27018 64-bit host=ip-10-169-132-210
      2014-11-19T16:21:58.488+0000 I CONTROL  [conn72] db version v2.8.0-rc1-pre-
      2014-11-19T16:21:58.488+0000 I CONTROL  [conn72] git version: b1be46d80c999a4a583dd55dff5da42b28480f15
      2014-11-19T16:21:58.488+0000 I CONTROL  [conn72] build info: Linux build19.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
      2014-11-19T16:21:58.488+0000 I CONTROL  [conn72] allocator: tcmalloc
      2014-11-19T16:21:58.488+0000 I CONTROL  [conn72] options: { config: "/data/mmsdev_4/automation-mongod.conf", net: { port: 27018 }, processManagement: { fork: true }, replication: { replSetName: "mmsdev" }, 
      storage: { dbPath: "/data/mmsdev_4", engine: "wiredtiger" }, systemLog: { destination: "file", path: "/data/mmsdev_4/mongodb.log" } }
      

      .....

      2014-11-19T16:32:22.764+0000 I INDEX    [rsSync]         building index using bulk method
      2014-11-19T16:32:22.952+0000 W STORAGE  [rsSync] failed to create WiredTiger bulk cursor: Device or resource busy
      2014-11-19T16:32:22.952+0000 W STORAGE  [rsSync] falling back to non-bulk cursor for index table:index-1065-2116325142371374767
      2014-11-19T16:32:22.953+0000 E STORAGE  [rsSync] WiredTiger (0) [1416414742:952730][20764:0x7f83e042e700], session.commit_transaction: failed transaction requires rollback
      2014-11-19T16:32:22.954+0000 I -        [rsSync] Fatal assertion 28519 BadValue 22: Invalid argument
      2014-11-19T16:32:22.977+0000 I CONTROL  [rsSync] 
       0xf73749 0xf1b791 0xf0144b 0xd93624 0xd9376a 0x90063c 0xa7aaf6 0xa79054 0x9179cc 0x91896f 0x923338 0x924949 0xcbbf88 0xcbcc22 0xcbd570 0xcc5d75 0xfc0d54 0x7f83e6c34182 0x7f83e5d34fbd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B73749"},{"b":"400000","o":"B1B791"},{"b":"400000","o":"B0144B"},{"b":"400000","o":"993624"},{"b":"400000","o":"99376A"},{"b":"400000","o":"50063C"},{"b":"400000","o":"67AAF
      6"},{"b":"400000","o":"679054"},{"b":"400000","o":"5179CC"},{"b":"400000","o":"51896F"},{"b":"400000","o":"523338"},{"b":"400000","o":"524949"},{"b":"400000","o":"8BBF88"},{"b":"400000","o":"8BCC22"},{"b":"
      400000","o":"8BD570"},{"b":"400000","o":"8C5D75"},{"b":"400000","o":"BC0D54"},{"b":"7F83E6C2C000","o":"8182"},{"b":"7F83E5C3A000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc1-pre-", "gitVersi
      on" : "b1be46d80c999a4a583dd55dff5da42b28480f15", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" 
      : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFE6CFE000", "elfType" : 3 }, { "b" : "7F83E6C2C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F83E6A24000", "path" : "/
      lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F83E6820000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F83E651C000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++
      .so.6", "elfType" : 3 }, { "b" : "7F83E6216000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F83E6000000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" :
       "7F83E5C3A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F83E6E4A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf73749]
       mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf1b791]
       mongod(_ZN5mongo23fassertFailedWithStatusEiRKNS_6StatusE+0xAB) [0xf0144b]
       mongod(_ZN5mongo22WiredTigerRecoveryUnit9_txnCloseEb+0x244) [0xd93624]
       mongod(_ZN5mongo22WiredTigerRecoveryUnit7_commitEv+0x8A) [0xd9376a]
       mongod(_ZN5mongo15WriteUnitOfWork6commitEv+0x1C) [0x90063c]
       mongod(_ZN5mongo26BtreeBasedBulkAccessMethod6commitEPSt3setINS_7DiskLocESt4lessIS2_ESaIS2_EEbb+0x106) [0xa7aaf6]
       mongod(_ZN5mongo22BtreeBasedAccessMethod10commitBulkEPNS_17IndexAccessMethodEbbPSt3setINS_7DiskLocESt4lessIS4_ESaIS4_EE+0x44) [0xa79054]
       mongod(_ZN5mongo15MultiIndexBlock13doneInsertingEPSt3setINS_7DiskLocESt4lessIS2_ESaIS2_EE+0x13C) [0x9179cc]
       mongod(_ZN5mongo15MultiIndexBlock30insertAllDocumentsInCollectionEPSt3setINS_7DiskLocESt4lessIS2_ESaIS2_EE+0x25F) [0x91896f]
       mongod(_ZN5mongo6Cloner11copyIndexesEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbb+0x458) [0x923338]
       mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0x11F9) [0x924949]
       mongod(+0x8BBF88) [0xcbbf88]
       mongod(+0x8BCC22) [0xcbcc22]
       mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x40) [0xcbd570]
       mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc5d75]
       mongod(+0xBC0D54) [0xfc0d54]
       libpthread.so.0(+0x8182) [0x7f83e6c34182]
       libc.so.6(clone+0x6D) [0x7f83e5d34fbd]
      -----  END BACKTRACE  -----
      2014-11-19T16:32:22.978+0000 I -        [rsSync] 
      
      ***aborting after fassert() failure
      
      
      2014-11-19T16:32:22.990+0000 F -        [rsSync] Got signal: 6 (Aborted).
      
       0xf73749 0xf730b2 0xf73436 0x7f83e5c70c30 0x7f83e5c70bb9 0x7f83e5c73fc8 0xf01490 0xd93624 0xd9376a 0x90063c 0xa7aaf6 0xa79054 0x9179cc 0x91896f 0x923338 0x924949 0xcbbf88 0xcbcc22 0xcbd570 0xcc5d75 0xfc0d54 0x7f83e6c34182 0x7f83e5d34fbd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B73749"},{"b":"400000","o":"B730B2"},{"b":"400000","o":"B73436"},{"b":"7F83E5C3A000","o":"36C30"},{"b":"7F83E5C3A000","o":"36BB9"},{"b":"7F83E5C3A000","o":"39FC8"},{"b":"400000","o":"B01490"},{"b":"400000","o":"993624"},{"b":"400000","o":"99376A"},{"b":"400000","o":"50063C"},{"b":"400000","o":"67AAF6"},{"b":"400000","o":"679054"},{"b":"400000","o":"5179CC"},{"b":"400000","o":"51896F"},{"b":"400000","o":"523338"},{"b":"400000","o":"524949"},{"b":"400000","o":"8BBF88"},{"b":"400000","o":"8BCC22"},{"b":"400000","o":"8BD570"},{"b":"400000","o":"8C5D75"},{"b":"400000","o":"BC0D54"},{"b":"7F83E6C2C000","o":"8182"},{"b":"7F83E5C3A000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc1-pre-", "gitVersion" : "b1be46d80c999a4a583dd55dff5da42b28480f15", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFE6CFE000", "elfType" : 3 }, { "b" : "7F83E6C2C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F83E6A24000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F83E6820000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F83E651C000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F83E6216000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F83E6000000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F83E5C3A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F83E6E4A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf73749]
       mongod(+0xB730B2) [0xf730b2]
       mongod(+0xB73436) [0xf73436]
       libc.so.6(+0x36C30) [0x7f83e5c70c30]
       libc.so.6(gsignal+0x39) [0x7f83e5c70bb9]
       libc.so.6(abort+0x148) [0x7f83e5c73fc8]
       mongod(_ZN5mongo23fassertFailedWithStatusEiRKNS_6StatusE+0xF0) [0xf01490]
       mongod(_ZN5mongo22WiredTigerRecoveryUnit9_txnCloseEb+0x244) [0xd93624]
       mongod(_ZN5mongo22WiredTigerRecoveryUnit7_commitEv+0x8A) [0xd9376a]
       mongod(_ZN5mongo15WriteUnitOfWork6commitEv+0x1C) [0x90063c]
       mongod(_ZN5mongo26BtreeBasedBulkAccessMethod6commitEPSt3setINS_7DiskLocESt4lessIS2_ESaIS2_EEbb+0x106) [0xa7aaf6]
       mongod(_ZN5mongo22BtreeBasedAccessMethod10commitBulkEPNS_17IndexAccessMethodEbbPSt3setINS_7DiskLocESt4lessIS4_ESaIS4_EE+0x44) [0xa79054]
       mongod(_ZN5mongo15MultiIndexBlock13doneInsertingEPSt3setINS_7DiskLocESt4lessIS2_ESaIS2_EE+0x13C) [0x9179cc]
       mongod(_ZN5mongo15MultiIndexBlock30insertAllDocumentsInCollectionEPSt3setINS_7DiskLocESt4lessIS2_ESaIS2_EE+0x25F) [0x91896f]
       mongod(_ZN5mongo6Cloner11copyIndexesEPNS_16OperationContextERKSsRKNS_15NamespaceStringES7_bbbbb+0x458) [0x923338]
       mongod(_ZN5mongo6Cloner2goEPNS_16OperationContextERKSsS4_RKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0x11F9) [0x924949]
       mongod(+0x8BBF88) [0xcbbf88]
       mongod(+0x8BCC22) [0xcbcc22]
       mongod(_ZN5mongo4repl17syncDoInitialSyncEv+0x40) [0xcbd570]
       mongod(_ZN5mongo4repl13runSyncThreadEv+0x185) [0xcc5d75]
       mongod(+0xBC0D54) [0xfc0d54]
       libpthread.so.0(+0x8182) [0x7f83e6c34182]
       libc.so.6(clone+0x6D) [0x7f83e5d34fbd]
      -----  END BACKTRACE  -----
      

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            kaloian.manassiev@mongodb.com Kaloian Manassiev
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: