Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3137

Hang in __log_slot_join/__log_slot_switch_internal

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • WT2.9.2, 3.2.13, 3.4.3, 3.5.4
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage 2017-01-23, Storage 2017-02-13

      • 24 cpus, 64 GB memory, 1 TB hard drive
      • ubuntu 15.10
      • mongod 3.4.0
      • standalone with: --wiredTigerCacheSizeGB 8 --wiredTigerCollectionBlockCompressor=zlib

      Run 20 threads of attached repro script with:

          for t in $(seq 20); do
              mongo --quiet test --eval "load('repro.js'); insert()" &
          done
          wait
      

      Script inserts 139 M generic text documents ranging from 1 kB to 4 MB according to a probability distribution encoded in the script, with smaller documents being much more common. Average document size is about 60 kB. Documents typically compress about 12:1 with zlib compression in the db.

      In three runs script

      • hung at ~35%
      • reached 50% without hanging when I terminated it
      • hung at about 50%

      When it hung mongod was consuming 2100% CPU, a majority of that system CPU. Per gdb stacks (attached) I think this is accounted for by 20 threads spinning in __log_slot_join and one thread spinning in __log_slot_switch_internal.

        1. repro-01-13-mongod.log
          155 kB
          Bruce Lucas
        2. repro-01-12-mongod.log
          154 kB
          Bruce Lucas
        3. log+gdb-2.tgz
          39 kB
          Bruce Lucas
        4. log+gdb.tgz
          30 kB
          Bruce Lucas
        5. r0.log
          205 kB
          Bruce Lucas
        6. stacks1.txt
          70 kB
          Bruce Lucas
        7. stacks2.txt
          70 kB
          Bruce Lucas
        8. stacks3.txt
          70 kB
          Bruce Lucas
        9. repro.js
          82 kB
          Bruce Lucas

            Assignee:
            sue.loverso@mongodb.com Susan LoVerso
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: