Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1404

Test/format seeing EBUSY return in verify

    Details

    • Type: Task
    • Status: Closed
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.5.2
    • Labels:

      Description

      There was a Jenkins test failure over night:
      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/2529/console

      The interesting output is:

      t: session.verify: file:wt: post-ops verify: Device or resource busy
      

      The particular config is:

      ############################################
      #  RUN PARAMETERS
      ############################################
      auto_throttle=1
      firstfit=0
      bitcnt=3
      bloom=0
      bloom_bit_count=21
      bloom_hash_count=31
      bloom_oldest=0
      cache=1
      checkpoints=1
      checksum=uncompressed
      chunk_size=6
      compaction=1
      compression=zlib
      data_extend=0
      data_source=file
      delete_pct=12
      dictionary=0
      evict_max=5
      file_type=variable-length column-store
      backups=0
      huffman_key=0
      huffman_value=0
      insert_pct=8
      internal_key_truncation=1
      internal_page_max=14
      isolation=read-committed
      key_gap=20
      key_max=39
      key_min=28
      leak_memory=0
      leaf_page_max=17
      logging=0
      lsm_worker_threads=4
      merge_max=8
      mmap=1
      ops=100000
      prefix_compression=1
      prefix_compression_min=8
      repeat_data_pct=64
      reverse=0
      rows=100000
      runs=1
      split_pct=57
      statistics=0
      threads=1
      timer=0
      value_max=268
      value_min=16
      wiredtiger_config=
      write_pct=57
      ############################################
      

      The console goes on to report memory leaks from Berkeley DB, but I suspect that is a result of exiting aggressively.

        Issue Links

          Activity

          Hide
          sueloverso Sue Loverso added a comment -

          And again today with sanitizer:
          http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/2970/

          CONFIG:

          ############################################
          #  RUN PARAMETERS
          ############################################
          abort=0
          auto_throttle=1
          firstfit=0
          bitcnt=4
          bloom=1
          bloom_bit_count=24
          bloom_hash_count=28
          bloom_oldest=0
          cache=76
          checkpoints=1
          checksum=off
          chunk_size=5
          compaction=0
          compression=snappy
          data_extend=0
          data_source=table
          delete_pct=13
          dictionary=0
          evict_max=4
          file_type=row-store
          backups=0
          huffman_key=0
          huffman_value=0
          insert_pct=30
          internal_key_truncation=1
          internal_page_max=10
          isolation=snapshot
          key_gap=14
          key_max=106
          key_min=31
          leak_memory=0
          leaf_page_max=14
          logging=1
          logging_archive=1
          logging_prealloc=1
          lsm_worker_threads=3
          merge_max=7
          mmap=1
          ops=100000
          prefix_compression=1
          prefix_compression_min=0
          repeat_data_pct=19
          reverse=0
          rows=100000
          runs=1
          split_pct=83
          statistics=0
          statistics_server=0
          threads=11
          timer=20
          value_max=1407
          value_min=17
          wiredtiger_config=
          write_pct=1
          ############################################
          

          Show
          sueloverso Sue Loverso added a comment - And again today with sanitizer: http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/2970/ CONFIG: ############################################ # RUN PARAMETERS ############################################ abort=0 auto_throttle=1 firstfit=0 bitcnt=4 bloom=1 bloom_bit_count=24 bloom_hash_count=28 bloom_oldest=0 cache=76 checkpoints=1 checksum=off chunk_size=5 compaction=0 compression=snappy data_extend=0 data_source=table delete_pct=13 dictionary=0 evict_max=4 file_type=row-store backups=0 huffman_key=0 huffman_value=0 insert_pct=30 internal_key_truncation=1 internal_page_max=10 isolation=snapshot key_gap=14 key_max=106 key_min=31 leak_memory=0 leaf_page_max=14 logging=1 logging_archive=1 logging_prealloc=1 lsm_worker_threads=3 merge_max=7 mmap=1 ops=100000 prefix_compression=1 prefix_compression_min=0 repeat_data_pct=19 reverse=0 rows=100000 runs=1 split_pct=83 statistics=0 statistics_server=0 threads=11 timer=20 value_max=1407 value_min=17 wiredtiger_config= write_pct=1 ############################################
          Hide
          agorrod Alex Gorrod added a comment -

          I've got some stack traces that I think indicate the issue:

          Thread 3 (Thread 0x7ffff5803700 (LWP 24183)):
          #0  0x00007ffff73fbbdf in pthread_cond_timedwait@@GLIBC_2.3.2 ()
             from /lib64/libpthread.so.0
          WT-1  0x0000000000438ebd in __wt_cond_wait (session=0x734c00, cond=0x74f670, 
              usecs=100000) at ../src/os_posix/os_mtx_cond.c:79
          WT-2  0x0000000000422baf in __evict_tree_walk_clear (session=0x734c00)
              at ../src/evict/evict_lru.c:619
          WT-3  0x0000000000422d6d in __wt_evict_file_exclusive_on (session=0x734c00, 
              evict_resetp=0x7ffff5802c44) at ../src/evict/evict_lru.c:700
          WT-4  0x00000000004c1bc0 in __wt_evict_file (session=0x734c00, syncop=2)
              at ../src/evict/evict_file.c:27
          WT-5  0x000000000049132f in __wt_cache_op (session=0x734c00, 
              ckptbase=0x7fffd80f6200, op=2) at ../src/btree/bt_sync.c:251
          WT-6  0x000000000046be94 in __checkpoint_worker (session=0x734c00, cfg=0x0, 
              is_checkpoint=0) at ../src/txn/txn_ckpt.c:888
          WT-7  0x000000000046c267 in __wt_checkpoint_close (session=0x734c00, force=0)
              at ../src/txn/txn_ckpt.c:1010
          WT-8  0x00000000004a3d56 in __wt_conn_btree_sync_and_close (session=0x734c00, 
              force=0) at ../src/conn/conn_dhandle.c:276
          WT-9  0x0000000000417ef0 in __sweep (session=0x734c00)
              at ../src/conn/conn_sweep.c:71
          WT-10 0x000000000041816f in __sweep_server (arg=0x734c00)
              at ../src/conn/conn_sweep.c:132
          WT-11 0x00007ffff73f7f18 in start_thread () from /lib64/libpthread.so.0
          WT-12 0x00007ffff6b01b2d in clone () from /lib64/libc.so.6
           
          Thread 1 (Thread 0x7ffff7fe6740 (LWP 24173)):
          #0  0x00007ffff6a52be9 in raise () from /lib64/libc.so.6
          ---Type <return> to continue, or q <return> to quit---
          WT-1  0x00007ffff6a53fe8 in abort () from /lib64/libc.so.6
          WT-2  0x00000000004ceca5 in __wt_abort (session=0x735180)
              at ../src/os_posix/os_abort.c:25
          WT-3  0x00000000004614c4 in __wt_assert (session=0x735180, error=0, 
              file_name=0x4fda0c "../src/schema/schema_worker.c", line_number=55, 
              fmt=0x4fda09 "%s") at ../src/support/err.c:464
          WT-4  0x0000000000458df4 in __wt_schema_worker (session=0x735180, 
              uri=0x72f510 "file:wt", file_func=0x4928f1 <__wt_verify>, name_func=0x0, 
              cfg=0x7fffffffdd80, open_flags=8196) at ../src/schema/schema_worker.c:55
          WT-5  0x000000000045d1a6 in __session_verify (wt_session=0x735180, 
              uri=0x72f510 "file:wt", config=0x0) at ../src/session/session_api.c:738
          WT-6  0x000000000040cae6 in wts_verify (tag=0x4f56ff "post-ops verify")
              at ../../../test/format/wts.c:457
          WT-7  0x00000000004097b1 in main (argc=0, argv=0x7fffffffdf60)
              at ../../../test/format/t.c:216
          
          

          Show
          agorrod Alex Gorrod added a comment - I've got some stack traces that I think indicate the issue: Thread 3 (Thread 0x7ffff5803700 (LWP 24183)): #0 0x00007ffff73fbbdf in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 WT-1 0x0000000000438ebd in __wt_cond_wait (session=0x734c00, cond=0x74f670, usecs=100000) at ../src/os_posix/os_mtx_cond.c:79 WT-2 0x0000000000422baf in __evict_tree_walk_clear (session=0x734c00) at ../src/evict/evict_lru.c:619 WT-3 0x0000000000422d6d in __wt_evict_file_exclusive_on (session=0x734c00, evict_resetp=0x7ffff5802c44) at ../src/evict/evict_lru.c:700 WT-4 0x00000000004c1bc0 in __wt_evict_file (session=0x734c00, syncop=2) at ../src/evict/evict_file.c:27 WT-5 0x000000000049132f in __wt_cache_op (session=0x734c00, ckptbase=0x7fffd80f6200, op=2) at ../src/btree/bt_sync.c:251 WT-6 0x000000000046be94 in __checkpoint_worker (session=0x734c00, cfg=0x0, is_checkpoint=0) at ../src/txn/txn_ckpt.c:888 WT-7 0x000000000046c267 in __wt_checkpoint_close (session=0x734c00, force=0) at ../src/txn/txn_ckpt.c:1010 WT-8 0x00000000004a3d56 in __wt_conn_btree_sync_and_close (session=0x734c00, force=0) at ../src/conn/conn_dhandle.c:276 WT-9 0x0000000000417ef0 in __sweep (session=0x734c00) at ../src/conn/conn_sweep.c:71 WT-10 0x000000000041816f in __sweep_server (arg=0x734c00) at ../src/conn/conn_sweep.c:132 WT-11 0x00007ffff73f7f18 in start_thread () from /lib64/libpthread.so.0 WT-12 0x00007ffff6b01b2d in clone () from /lib64/libc.so.6   Thread 1 (Thread 0x7ffff7fe6740 (LWP 24173)): #0 0x00007ffff6a52be9 in raise () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- WT-1 0x00007ffff6a53fe8 in abort () from /lib64/libc.so.6 WT-2 0x00000000004ceca5 in __wt_abort (session=0x735180) at ../src/os_posix/os_abort.c:25 WT-3 0x00000000004614c4 in __wt_assert (session=0x735180, error=0, file_name=0x4fda0c "../src/schema/schema_worker.c", line_number=55, fmt=0x4fda09 "%s") at ../src/support/err.c:464 WT-4 0x0000000000458df4 in __wt_schema_worker (session=0x735180, uri=0x72f510 "file:wt", file_func=0x4928f1 <__wt_verify>, name_func=0x0, cfg=0x7fffffffdd80, open_flags=8196) at ../src/schema/schema_worker.c:55 WT-5 0x000000000045d1a6 in __session_verify (wt_session=0x735180, uri=0x72f510 "file:wt", config=0x0) at ../src/session/session_api.c:738 WT-6 0x000000000040cae6 in wts_verify (tag=0x4f56ff "post-ops verify") at ../../../test/format/wts.c:457 WT-7 0x00000000004097b1 in main (argc=0, argv=0x7fffffffdf60) at ../../../test/format/t.c:216
          Hide
          agorrod Alex Gorrod added a comment -

          These issues are fixed by WT-1767 - close. If we see more test/format failures with EBUSY we can open a new issue.

          Show
          agorrod Alex Gorrod added a comment - These issues are fixed by WT-1767 - close. If we see more test/format failures with EBUSY we can open a new issue.
          Hide
          michael.cahill Michael Cahill added a comment -

          See also BF-741, SERVER-16457, SERVER-16804 and SERVER-16869. I'll update any that are still open.

          Show
          michael.cahill Michael Cahill added a comment - See also BF-741, SERVER-16457 , SERVER-16804 and SERVER-16869 . I'll update any that are still open.
          Hide
          ramon.fernandez Ramon Fernandez added a comment -
          Show
          ramon.fernandez Ramon Fernandez added a comment - Additional ticket information from GitHub This ticket was referenced in the following commits: 4df72e8e20139ddf667e1f0d3b6b7dcf91deb006 0eecd0a2d97771380ecbd7fd27bd44988db1148c a9f0e3ac769a060a8c3b06bac71fbed0e5f46cc6 3188352d623de85803db3dd6e5b5188822e2db4f

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 48 weeks, 6 days ago
                Date of 1st Reply: