Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.7.0
    • Labels:
      None
    • # Replies:
      5
    • Last comment by Customer:
      true

      Description

      The Jenkins python long unit test hung:
      http://build.wiredtiger.com:8080/job/wiredtiger-test-unit-long/904/

      The call stack is:

      (gdb) where
      #0  __wt_tree_walk (session=0x4c478d0, refp=0x7ffd900e2608, walkcntp=0x0,
          flags=13) at ../src/btree/bt_walk.c:364
      #1  0x00007f3e539c1fda in __wt_evict_file (session=0x4c478d0, syncop=2)
          at ../src/evict/evict_file.c:34
      #2  0x00007f3e5397238a in __wt_cache_op (session=0x4c478d0,
          ckptbase=0x131935a0, op=2) at ../src/btree/bt_sync.c:262
      #3  0x00007f3e53a32272 in __checkpoint_worker (session=0x4c478d0, cfg=0x0,
          is_checkpoint=false, need_tracking=false) at ../src/txn/txn_ckpt.c:1102
      #4  0x00007f3e53a3287b in __wt_checkpoint_close (session=0x4c478d0,
          final=false) at ../src/txn/txn_ckpt.c:1271
      #5  0x00007f3e539931c5 in __wt_conn_btree_sync_and_close (session=0x4c478d0,
          final=false, force=false) at ../src/conn/conn_dhandle.c:199
      #6  0x00007f3e5399400c in __wt_conn_dhandle_close_all (session=0x4c478d0,
          uri=0x7f3e50dbc08c "file:test_truncate", force=false)
          at ../src/conn/conn_dhandle.c:570
      #7  0x00007f3e53a0d5ba in __drop_file (session=0x4c478d0,
          uri=0x7f3e50dbc08c "file:test_truncate", force=false, cfg=0x7ffd900e29f0)
          at ../src/schema/schema_drop.c:32
      #8  0x00007f3e53a0dc2c in __wt_schema_drop (session=0x4c478d0,
          uri=0x7f3e50dbc08c "file:test_truncate", cfg=0x7ffd900e29f0)
          at ../src/schema/schema_drop.c:180
      #9  0x00007f3e53a1cc32 in __wt_session_drop (session=0x4c478d0,
          uri=0x7f3e50dbc08c "file:test_truncate", cfg=0x7ffd900e29f0)
          at ../src/session/session_api.c:587
      #10 0x00007f3e53a1ce1f in __session_drop (wt_session=0x4c478d0,
          uri=0x7f3e50dbc08c "file:test_truncate", config=0x0)
          at ../src/session/session_api.c:609
      #11 0x00007f3e53c87880 in _wrap_Session_drop (self=<optimized out>,
          args=<optimized out>) at wiredtiger_wrap.c:6355
      

      There are no other active threads. The issue is that the __wt_tree_walk code is constantly re-starting it's search due to the root page containing a number of refs in the WT_REF_SPLIT state:

      (gdb) p *$14
      $15 = {entries = 11, deleted_entries = 2, index = 0x2ea6960}
      (gdb) p *$14->index[0]
      $18 = {page = 0x0, home = 0x290b0d0, pindex_hint = 0, state = 5, addr = 0x0,
        key = {recno = 4294967379, ikey = 0x100000053}, page_del = 0x0}
      (gdb) p *$14->index[1]
      $20 = {page = 0x0, home = 0x290b0d0, pindex_hint = 1, state = 5,
        addr = 0x4cb5525, key = {recno = 4294967401, ikey = 0x100000069},
        page_del = 0x13177a30}
      (gdb) p *$14->index[2]
      $21 = {page = 0x0, home = 0x290b0d0, pindex_hint = 2, state = 5,
        addr = 0x4cb5530, key = {recno = 4294967423, ikey = 0x10000007f},
        page_del = 0x22ba750}
      (gdb) p *$14->index[3]
      $22 = {page = 0x0, home = 0x290b0d0, pindex_hint = 3, state = 5,
        addr = 0x4cb553b, key = {recno = 4294967445, ikey = 0x100000095},
        page_del = 0x2184220}
      (gdb) p *$14->index[4]
      $23 = {page = 0x0, home = 0x290b0d0, pindex_hint = 4, state = 5,
        addr = 0x4cb5546, key = {recno = 4294967467, ikey = 0x1000000ab},
        page_del = 0x1f451c0}
      (gdb) p *$14->index[5]
      $24 = {page = 0x0, home = 0x290b0d0, pindex_hint = 5, state = 5,
        addr = 0x4cb5551, key = {recno = 4294967489, ikey = 0x1000000c1},
        page_del = 0x43e8d30}
      (gdb) p *$14->index[6]
      $25 = {page = 0x0, home = 0x290b0d0, pindex_hint = 6, state = 5,
        addr = 0x4cb555c, key = {recno = 4294967511, ikey = 0x1000000d7},
        page_del = 0x2083e20}
      (gdb) p *$14->index[7]
      $26 = {page = 0x0, home = 0x290b0d0, pindex_hint = 7, state = 5,
        addr = 0x4cb5568, key = {recno = 8589934829, ikey = 0x2000000ed},
        page_del = 0x1eee330}
      (gdb) p *$14->index[8]
      $27 = {page = 0x0, home = 0x290b0d0, pindex_hint = 8, state = 5,
        addr = 0x4cb5574, key = {recno = 8589934853, ikey = 0x200000105},
        page_del = 0x2e36f50}
      (gdb) p *$14->index[9]
      $28 = {page = 0x0, home = 0x290b0d0, pindex_hint = 9, state = 5,
        addr = 0x4cb557f, key = {recno = 8589934875, ikey = 0x20000011b},
        page_del = 0x1d597d0}
      (gdb) p *$14->index[10]
      $29 = {page = 0x0, home = 0x290b0d0, pindex_hint = 10, state = 2, addr = 0x0,
        key = {recno = 8589934899, ikey = 0x200000133}, page_del = 0x0}
      

      I haven't attempted to reproduce yet. I'll try that next.

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-2195 Fix a hang after giving up on a reverse split.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/423cb8827a88dd2d4ec68a46e237d8710fb9cf3a

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2195 Fix a hang after giving up on a reverse split. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/423cb8827a88dd2d4ec68a46e237d8710fb9cf3a
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2285 from wiredtiger/WT-2195

          WT-2195 Fix a hang after giving up on a reverse split.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/438f455407f2c47ff68fbbe7efa707b5f0c54712

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2285 from wiredtiger/ WT-2195 WT-2195 Fix a hang after giving up on a reverse split. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/438f455407f2c47ff68fbbe7efa707b5f0c54712
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2285 from wiredtiger/WT-2195

          WT-2195 Fix a hang after giving up on a reverse split.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/438f455407f2c47ff68fbbe7efa707b5f0c54712

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2285 from wiredtiger/ WT-2195 WT-2195 Fix a hang after giving up on a reverse split. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/438f455407f2c47ff68fbbe7efa707b5f0c54712
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-2195 Fix a hang after giving up on a reverse split.
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/66a111ec48da60195a011e2a163bef07f5035bb0

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2195 Fix a hang after giving up on a reverse split. Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/66a111ec48da60195a011e2a163bef07f5035bb0
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

          Message: Import wiredtiger-wiredtiger-mongodb-3.0.7-9-gdeb2d81.tar.gz from wiredtiger branch mongodb-3.0

          ref: cb64236..deb2d81

          deb2d81 SERVER-21027 Reverse split if there are many deleted pages (3.0)
          66a111e WT-2195 Fix a hang after giving up on a reverse split.
          7b1398a SERVER-21027 Don't leave empty internal pages in the tree
          c819d2f SERVER-21027 Fix reverse splits to keep the original child ref locked
          00dfebc SERVER-21027 Reverse split if there are many deleted pages
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/9add8acc69a119949a156b815003ecc15db75e0d

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger-wiredtiger-mongodb-3.0.7-9-gdeb2d81.tar.gz from wiredtiger branch mongodb-3.0 ref: cb64236..deb2d81 deb2d81 SERVER-21027 Reverse split if there are many deleted pages (3.0) 66a111e WT-2195 Fix a hang after giving up on a reverse split. 7b1398a SERVER-21027 Don't leave empty internal pages in the tree c819d2f SERVER-21027 Fix reverse splits to keep the original child ref locked 00dfebc SERVER-21027 Reverse split if there are many deleted pages Branch: v3.0 https://github.com/mongodb/mongo/commit/9add8acc69a119949a156b815003ecc15db75e0d

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 29 weeks, 4 days ago
                Date of 1st Reply: