Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1975

Stress test assertion failure in logging

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.7.0
    • Labels:
      None
    • # Replies:
      6
    • Last comment by Customer:
      true

      Description

      test-format-stress-sanitizer failed with this assertion from the log-server:

      [1435020349:94772][23207:0x7f52497fc700], t, log-server: ../src/os_posix/os_remove.c, 37: fh == NULL
      [1435020349:94837][23207:0x7f52497fc700], t, log-server: aborting WiredTiger library
      

        Issue Links

          Activity

          Hide
          sue.loverso Sue LoVerso added a comment -

          Here is the failing CONFIG:

          ############################################
          #  RUN PARAMETERS
          ############################################
          abort=0
          auto_throttle=1
          firstfit=0
          bitcnt=1
          bloom=1
          bloom_bit_count=55
          bloom_hash_count=4
          bloom_oldest=0
          cache=87
          checkpoints=1
          checksum=uncompressed
          chunk_size=2
          compaction=0
          compression=snappy
          data_extend=0
          data_source=table
          delete_pct=13
          dictionary=0
          encryption=none
          evict_max=4
          file_type=variable-length column-store
          backups=0
          huffman_key=0
          huffman_value=0
          insert_pct=27
          internal_key_truncation=1
          internal_page_max=13
          isolation=read-committed
          key_gap=9
          key_max=47
          key_min=24
          leak_memory=0
          leaf_page_max=16
          logging=1
          logging_compression=none
          logging_archive=1
          logging_prealloc=0
          lsm_worker_threads=3
          merge_max=18
          mmap=1
          ops=100000
          prefix_compression=1
          prefix_compression_min=7
          repeat_data_pct=89
          reverse=0
          rows=100000
          runs=1
          split_pct=49
          statistics=0
          statistics_server=0
          threads=9
          timer=20
          value_max=42
          value_min=17
          wiredtiger_config=
          write_pct=12
          ############################################
          

          Since sanitizer does not generate core files, I'll try to reproduce this. Looks like it tried to archive a log file that was still open.

          Show
          sue.loverso Sue LoVerso added a comment - Here is the failing CONFIG: ############################################ # RUN PARAMETERS ############################################ abort=0 auto_throttle=1 firstfit=0 bitcnt=1 bloom=1 bloom_bit_count=55 bloom_hash_count=4 bloom_oldest=0 cache=87 checkpoints=1 checksum=uncompressed chunk_size=2 compaction=0 compression=snappy data_extend=0 data_source=table delete_pct=13 dictionary=0 encryption=none evict_max=4 file_type=variable-length column-store backups=0 huffman_key=0 huffman_value=0 insert_pct=27 internal_key_truncation=1 internal_page_max=13 isolation=read-committed key_gap=9 key_max=47 key_min=24 leak_memory=0 leaf_page_max=16 logging=1 logging_compression=none logging_archive=1 logging_prealloc=0 lsm_worker_threads=3 merge_max=18 mmap=1 ops=100000 prefix_compression=1 prefix_compression_min=7 repeat_data_pct=89 reverse=0 rows=100000 runs=1 split_pct=49 statistics=0 statistics_server=0 threads=9 timer=20 value_max=42 value_min=17 wiredtiger_config= write_pct=12 ############################################ Since sanitizer does not generate core files, I'll try to reproduce this. Looks like it tried to archive a log file that was still open.
          Hide
          sue.loverso Sue LoVerso added a comment -

          It is unfortunate sanitizer does not generate core files. I've run 1000 iterations of this config successfully. I'll bump it up to 10k and study the code.

          Show
          sue.loverso Sue LoVerso added a comment - It is unfortunate sanitizer does not generate core files. I've run 1000 iterations of this config successfully. I'll bump it up to 10k and study the code.
          Hide
          sue.loverso Sue LoVerso added a comment -

          Although never able to repro this assertion, I found a code path with a window that could incorrectly move sync_lsn forward during a checkpoint and allow an in-use file to be considered available for archiving. Testing fix now.

          Show
          sue.loverso Sue LoVerso added a comment - Although never able to repro this assertion, I found a code path with a window that could incorrectly move sync_lsn forward during a checkpoint and allow an in-use file to be considered available for archiving. Testing fix now.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

          Message: WT-1975 Wait for sync_lsn of earlier log files to complete for forced
          sync.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/97b9e03319aa38271b2f41a3cb7084feba4d2b1c

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-1975 Wait for sync_lsn of earlier log files to complete for forced sync. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/97b9e03319aa38271b2f41a3cb7084feba4d2b1c
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Merge pull request #2026 from wiredtiger/sync-lsn-fixes

          WT-1975 Wait for sync_lsn of earlier log files to complete for forced sync.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/76d2e7369fb28e5af1a7f446d97daf37a3f2544c

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2026 from wiredtiger/sync-lsn-fixes WT-1975 Wait for sync_lsn of earlier log files to complete for forced sync. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/76d2e7369fb28e5af1a7f446d97daf37a3f2544c
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-1975 Wait for sync_lsn of earlier log files to complete for forced sync.

          (cherry picked from commit 76d2e7369fb28e5af1a7f446d97daf37a3f2544c)
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/6a667b9b0f00fcbf4e03360f2d9fd6c7a08abdb6

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-1975 Wait for sync_lsn of earlier log files to complete for forced sync. (cherry picked from commit 76d2e7369fb28e5af1a7f446d97daf37a3f2544c) Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/6a667b9b0f00fcbf4e03360f2d9fd6c7a08abdb6

            People

            • Assignee:
              sue.loverso Sue LoVerso
              Reporter:
              sue.loverso Sue LoVerso
              Participants:
              Last commenter:
              Michael Cahill
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 38 weeks, 3 days ago
                Date of 1st Reply: