Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2102

log-append hanging occasionally

        Details

        • Type: Bug
        • Status: Resolved
        • Priority: Major - P3
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: WT2.7.0
        • Labels:
          None

          Description

          The Jenkins run of log-append.wtperf hung with all relevant threads yielding in log_slot_join but no other thread in mid-operation.

            Issue Links

            is depended on by

            Improvement - An improvement or enhancement to an existing feature or task. SERVER-20408 WiredTiger changes for MongoDB 3.1.9

            • Major - P3 - Major loss of function.
            • Closed

              Activity

              • Comments
              Ascending order - Click to sort in descending order
              Hide
              Permalink
              sue.loverso Sue LoVerso added a comment -

              Unfortunately the Jenkins executable does not have symbols in it so other than stacks I cannot see any content. I'm running it on my AWS HDD box in a loop to try to repro.

              This must be highly optimized because _wt_log_write_internal is the caller of _wt_log_slot_join and it is not present on the stack.

              Thread 8 (Thread 0x7f268d7f9700 (LWP 8452)):
              		 
              #0  0x00007f2693e1bb97 in sched_yield () from /lib64/libc.so.6
              		 
              #1  0x00000000004249c8 in __wt_log_slot_join ()
              		 
              #2  0x0000000000422219 in __wt_log_write ()
              		 
              #3  0x000000000045cf18 in __wt_txn_commit ()
              		 
              #4  0x0000000000409155 in populate_thread ()

              Show
              sue.loverso Sue LoVerso added a comment - Unfortunately the Jenkins executable does not have symbols in it so other than stacks I cannot see any content. I'm running it on my AWS HDD box in a loop to try to repro. This must be highly optimized because _ wt_log_write_internal is the caller of _wt_log_slot_join and it is not present on the stack. Thread 8 (Thread 0x7f268d7f9700 (LWP 8452)): #0 0x00007f2693e1bb97 in sched_yield () from /lib64/libc.so.6 #1 0x00000000004249c8 in __wt_log_slot_join () #2 0x0000000000422219 in __wt_log_write () #3 0x000000000045cf18 in __wt_txn_commit () #4 0x0000000000409155 in populate_thread ()
              Hide
              Permalink
              sue.loverso Sue LoVerso added a comment -

              Good news is I can reproduce this every 5 or 6 iterations. We close and write a slot, but a new slot is not installed.

              Show
              sue.loverso Sue LoVerso added a comment - Good news is I can reproduce this every 5 or 6 iterations. We close and write a slot, but a new slot is not installed.
              Hide
              Permalink
              sue.loverso Sue LoVerso added a comment -

              I understand this now. Similar to log switching if a forced write hits the EBUSY from newfile having to wait for the previous log file to close, it needs to retry. I'm going to look at combining the internals from switch and force write as they're very similar so that we only have to solve problems like this in one place.

              Show
              sue.loverso Sue LoVerso added a comment - I understand this now. Similar to log switching if a forced write hits the EBUSY from newfile having to wait for the previous log file to close, it needs to retry. I'm going to look at combining the internals from switch and force write as they're very similar so that we only have to solve problems like this in one place.
              Hide
              Permalink
              xgen-internal-githook Githook User added a comment -

              Author:

              {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

              Message: WT-2102 Combine the switch and force_write functions.
              All paths check whether they have closed the slot and need to retry
              creating a new slot if they get EBUSY.
              Branch: develop
              https://github.com/wiredtiger/wiredtiger/commit/b833027ca90de4a3d0e64ec5761df22317913d87

              Show
              xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-2102 Combine the switch and force_write functions. All paths check whether they have closed the slot and need to retry creating a new slot if they get EBUSY. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/b833027ca90de4a3d0e64ec5761df22317913d87
              Hide
              Permalink
              xgen-internal-githook Githook User added a comment -

              Author:

              {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

              Message: Merge pull request #2186 from wiredtiger/wt-2102

              WT-2102 Combine the switch and force_write functions.
              Branch: develop
              https://github.com/wiredtiger/wiredtiger/commit/e1d6886824058b333495236b776b10fcd8fb74ae

              Show
              xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2186 from wiredtiger/wt-2102 WT-2102 Combine the switch and force_write functions. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/e1d6886824058b333495236b776b10fcd8fb74ae

                People

                • Assignee:
                  sue.loverso Sue LoVerso
                  Reporter:
                  sue.loverso Sue LoVerso
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: