Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3124

Unknown recovery failure on wiredtiger-test-check-long

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Trivial - P5 Trivial - P5
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None

      The following failure occured as part of this job:
      http://build.wiredtiger.com:8080/job/wiredtiger-test-check-long/337

      It occurred on tinderbox and is possibly due to the disk on that server being filled. Unfortunately, the error log is missing from the run's tarball

      GDB Trace, shows that there was an abort in the call to fclose, not 100% certain if this is related to a full disk.

      (gdb) thread apply all bt
      
      Thread 6 (Thread 0x7f98691b2700 (LWP 78493)):
      #0  0x0000003e6ce0c8e9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007f986a2c3e35 in __wt_vunpack_negint (pp=0x1b34f20, maxlen=30122176, retp=0x23668) at ../src/include/intpack.i:176
      #2  0x00007f986a2ffba5 in __wt_vfprintf (session=0x17318, fstr=0x1b248a8, fmt=0x1b34f20 "\020\340\261\001", ap=0x1cba0c0) at ../src/include/os_fstream.i:51
      #3  0x00007f986a257257 in __wt_spin_lock_track (session=0x2b995a3e, t=0x58738294) at ../src/include/mutex.i:295
      #4  0x0000003e6ce07555 in start_thread () from /lib64/libpthread.so.0
      #5  0x0000003e6cb02ded in clone () from /lib64/libc.so.6
      
      Thread 5 (Thread 0x7f98681b0700 (LWP 78496)):
      #0  0x0000003e6ce0c8e9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007f986a2c3e35 in __wt_vunpack_negint (pp=0x1b36210, maxlen=30300192, retp=0x989680) at ../src/include/intpack.i:176
      #2  0x00007f986a25b43d in __wt_curbackup_open (session=0x989680, uri=0x0, cfg=0x7f986a25b43d <__wt_curbackup_open+243>, cursorp=0x7f98681afef0) at ../src/cursor/cur_backup.c:136
      #3  0x00007f986a25bd00 in __backup_start (session=0x1b36210, cb=0x0, cfg=0x7f986a25bd00 <__backup_start+993>) at ../src/cursor/cur_backup.c:298
      #4  0x0000003e6ce07555 in start_thread () from /lib64/libpthread.so.0
      #5  0x0000003e6cb02ded in clone () from /lib64/libc.so.6
      
      Thread 4 (Thread 0x7f98699b3700 (LWP 78492)):
      #0  0x0000003e6ce0c8e9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007f986a2c3e35 in __wt_vunpack_negint (pp=0x1b34bf8, maxlen=30073936, retp=0x1a9c8) at ../src/include/intpack.i:176
      #2  0x00007f986a2ffba5 in __wt_vfprintf (session=0x182b8, fstr=0x0, fmt=0x1b34bf8 "\020\340\261\001", ap=0x1cae450) at ../src/include/os_fstream.i:51
      #3  0x00007f986a2ffc60 in __wt_fprintf (session=0x7f98699b3700, fstr=0x800000, fmt=0x0) at ../src/include/os_fstream.i:77
      #4  0x00007f986a257033 in __wt_scr_free (session=0x1b1e010, bufp=0x1b64000) at ../src/include/buf.i:121
      #5  0x0000003e6ce07555 in start_thread () from /lib64/libpthread.so.0
      #6  0x0000003e6cb02ded in clone () from /lib64/libc.so.6
      
      Thread 3 (Thread 0x7f98689b1700 (LWP 78495)):
      #0  0x0000003e6ce0c8e9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007f986a2c3e35 in __wt_vunpack_negint (pp=0x1b35570, maxlen=28512208, retp=0x1a9c8) at ../src/include/intpack.i:176
      #2  0x00007f986a2ffba5 in __wt_vfprintf (session=0x182b8, fstr=0x7f98689b0ef3, fmt=0x1b35570 "\020\340\261\001", ap=0x1b30fd0) at ../src/include/os_fstream.i:51
      #3  0x00007f986a2ffc60 in __wt_fprintf (session=0x7f98689b1700, fstr=0x800000, fmt=0x0) at ../src/include/os_fstream.i:77
      #4  0x00007f986a28c332 in __evict_tune_workers (session=0x7f986a2ffc60 <__wt_fprintf+158>) at ../src/evict/evict_lru.c:953
      #5  0x00007f986a30ca3d in __thread_group_resize (session=0x0, group=0x1b1e010, new_min=0, new_max=28511344, flags=0) at ../src/support/thread_group.c:146
      #6  0x0000003e6ce07555 in start_thread () from /lib64/libpthread.so.0
      #7  0x0000003e6cb02ded in clone () from /lib64/libc.so.6
      
      Thread 2 (Thread 0x7f986a1b4700 (LWP 78491)):
      #0  0x0000003e6ce0c8e9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007f986a2c3e35 in __wt_vunpack_negint (pp=0x1b348d0, maxlen=30073792, retp=0x186a0) at ../src/include/intpack.i:176
      #2  0x00007f986a255285 in __log_server (arg=0x1b348d0) at ../src/conn/conn_log.c:798
      #3  0x00007f986a2567ec in __wt_connection_close (conn=0x31b0a6b5) at ../src/conn/conn_open.c:154
      #4  0x0000003e6ce07555 in start_thread () from /lib64/libpthread.so.0
      #5  0x0000003e6cb02ded in clone () from /lib64/libc.so.6
      
      Thread 1 (Thread 0x7f986a1b6700 (LWP 78481)):
      #0  0x0000003e6ca349c8 in raise () from /lib64/libc.so.6
      #1  0x0000003e6ca3665a in abort () from /lib64/libc.so.6
      #2  0x000000000040183f in fill_db () at ../../../test/recovery/truncated-log.c:254
      #3  0x0000000000401954 in main (argc=0, argv=0x7ffc7c02a190) at ../../../test/recovery/truncated-log.c:305
      (gdb)
      

      Repro on a different server, and on the same server with the same binaries did not reproduce.

      Action here is to wait for a week or so, to confirm that the issue has now shown up again.

            Assignee:
            david.hows David Hows
            Reporter:
            david.hows David Hows
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: