Thanks David Hows. I'll take this.

The WT_CACHE_PASS_INTERRUPT is supposed to make the eviction server interrupt its pass and get back out to release that lock. I suspect what happened is that two threads set the flag, one cleared it and now we're in the deadlock because the server thread doesn't know there's a thread there waiting for the eviction pass to get interrupted. That flag may need to be a ref count.

Author:

Message: WT-2698 Change flag into atomically manipulated field to avoid deadlock. (#2796)
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/66a13643f484caedf497c67572f3ab73fa603f64

Author:

Message: WT-2698 Change flag into atomically manipulated field to avoid deadlock. (#2796)
Branch: mongodb-3.4
https://github.com/wiredtiger/wiredtiger/commit/66a13643f484caedf497c67572f3ab73fa603f64

Author:

Message: Import wiredtiger-wiredtiger-2.8.0-511-gd8fb874.tar.gz from wiredtiger branch mongodb-3.4

ref: fb1663e..d8fb874

SERVER-24580 Performance is poor when WiredTiger cache is full
WT-2319 Add statistics around fsync calls
WT-2408 Windows error translation layer
WT-2514 log path name is an empty string.
WT-2541 Add statistics for number of threads currently in read/write
WT-2605 C tests focused on different schema types needed for join and other APIs
WT-2622 unit test failed in test_cursor_random.test_cursor_random.test_cursor_random_multiple_page_records
WT-2652 Logging code improvements
WT-2664 Change eviction so any eviction thread can find candidates
WT-2667 Enhance WiredTiger Evergreen testing
WT-2668 Create join statistics that are useful and are easy to understand
WT-2671 dump more information about the file layout in verify debug mode
WT-2672 Handle system calls that don't set errno
WT-2676 Don't use key size in column store in-memory splits.
WT-2677 Fix JSON output so only printable ASCII is produced (seen on Solaris)
WT-2682 add option to configure WiredTiger with strict compiler flags
WT-2683 WiredTiger no longer needs to return non-zero disk sizes
WT-2685 hazard pointer failure from clear walk
WT-2686 logging subsystem core dump
WT-2687 test suite should verify the exit status of the wt utility
WT-2689 Use after free in WT_SESSION::open_cursor
WT-2691 Use wrappers for ctype functions to avoid sign extension errors
WT-2692 Fix race in file system example
WT-2696 Missing log records with large updates
WT-2698 Test/recovery hung
WT-2702 Under high thread load, WiredTiger exceeds cache size
WT-2704 test/format hung on bengal
WT-2706 Test failure running random-abort recovery test
WT-2707 dist/s_label enhancements, and error jump cleanups
WT-2709 connection reconfigure segfault in __wt_conn_cache_pool_destroy
WT-2710 WT_FILE_HANDLE_INMEM no longer needs an off field
WT-2712 Coverity 1356928 and 1356929: ASSERT_SIDE_EFFECT
WT-2713 document WT_PANIC so pluggable filesystems can panic.
WT-2714 lint
WT-2715 random-abort test may write partial record at the end
WT-2720 Pull request tester not running Python suite
WT-2722 s_label or s_label_loop false positive
WT-2724 Eviction workers created on open exit immediately
SERVER-24425 WiredTiger changes for MongoDB 3.3.9
Branch: master
https://github.com/mongodb/mongo/commit/7e3ce9cec306df5abb15c0c1a599023270cfb807

Author:

Message: WT-2698 Change flag into atomically manipulated field to avoid deadlock. (#2796)
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/66a13643f484caedf497c67572f3ab73fa603f64

Author:

Message: Import wiredtiger-wiredtiger-2.8.0-592-g848e5f5.tar.gz from wiredtiger branch mongodb-3.2

ref: 8b7110b..848e5f5
for: 3.2.9

This commit replaces a number of previous backports with the original
change, so some of the tickets listed below may have been resolved in
previous versions already as a backport to the v3.2 branch. For example,
SERVER-24580 includes change a63e21b8 as a cherry-pick of 1f4aaa44, but
this commit replaces that change with 1f4aaa44 itself.

SERVER-23659 Provide useful message when wiredTigerJournalCompressor is changed
SERVER-23661 $sample takes disproportionately long time on newly created collection
SERVER-24306 40-second journaling stall from "log files prepared" to checkpoint
SERVER-24580 Improve performance when WiredTiger cache is full
WT-2103 Add incremental backup testing to format
WT-2223 Add stress testing for in-memory
WT-2268 JSON load incorrect with UNICODE input
WT-2319 Add statistics around fsync calls
WT-2325 Fix an incomplete comment
WT-2343 Assert we don't remove or rename when backup cursor is open
WT-2349 Add ability to open databases read-only
WT-2359 WiredTiger with Python will hang if a calloc failure occurs during __wt_connection_close
WT-2360 Allow disjunctions and combinations of operations in join cursors
WT-2408 Windows error translation layer
WT-2446 Estimate WT cache hit ratio
WT-2450 Salvage releases pages, then explicitly evicts them.
WT-2453 Throughput drop in wtperf evict Jenkins tests
WT-2479 Dump utility discards table config (JSON)
WT-2491 The dhandle close_lock isn't valuable at the moment
WT-2504 Should READONLY always read basecfg file?
WT-2505 Review clang analyzer warnings
WT-2508 Test programs should remove test directories on the "clean" target
WT-2514 Log path name is an empty string.
WT-2518 LSM checkpoint handle acquisition optimization
WT-2520 WT_SESSION::verify should not alter tables
WT-2526 Mixing and matching readonly and read/write handles
WT-2535 Extend test/format to test for transactions reading their writes
WT-2537 Cannot open DB written by WT2.6.1 with WT2.8.0 due to WT_NOTFOUND on recovery
WT-2539 Implement file streaming above pluggable filesystems
WT-2540 Separate stream and file handle methods
WT-2541 Add statistics for number of threads currently in read/write
WT-2542 Fixed-length column store reconciliation overwrites original values
WT-2544 Fix eviction statistics when clear is configured
WT-2546 Eviction server not help evict pages sometimes
WT-2547 Add 1-eviction-worker jobs to Jenkins
WT-2548 Cap the amount of data handed to raw compression.
WT-2549 joins using recno keys return no values
WT-2550 java ex_schema example fails
WT-2552 Public API for pluggable filesystems
WT-2553 Document in-memory configuration and WT_CACHE_FULL error return
WT-2554 Implement a framework for adding C test cases
WT-2556 Typo in the Java example code
WT-2557 format test program should discard log files after incremental backup
WT-2558 WT_PAGE structure reorganization
WT-2559 Windows segfault in logging code
WT-2560 Stuck trying to update oldest transaction ID
WT-2562 Reconfig02 test failing sometimes on PPC
WT-2565 Item 3573 on page at [write-check] is a corrupted cell
WT-2566 All lock operations should be barriers
WT-2567 Segfault in test/format log truncate
WT-2568 Java PackTest.java compilation error
WT-2569 Win_handle_read should always call GetLastError on error
WT-2570 Minor lint cleanups.
WT-2571 Join code cleanup
WT-2572 Don't select an in-memory format run if incompatible options configured
WT-2573 Free of stack-allocated WT_REF
WT-2574 Format doesn't free all allocated configure memory
WT-2576 Variable-length column-store out-of-order return
WT-2577 Core dump discarding non-existent addresses
WT-2579 In-memory configurations break debugging support
WT-2580 Potential SWIG naming conflict in Java
WT-2581 Assert multi->disk_image == NULL
WT-2582 Cache eviction server error: WT_RESTART
WT-2583 Incremental backup can prevent future recovery
WT-2584 Don't use periods in error messages
WT-2586 Remove ex_config.c until config cursors are supported
WT-2589 Check stats using WT_STAT_SET and 'clear' usage
WT-2592 Joins using non-recno key types not working
WT-2593 Disk full with pre-allocated log files
WT-2595 Fix compiler warning in packing tests
WT-2597 JSON dump needs LSM tests
WT-2598 In-memory FS needs fast lookup on file names
WT-2599 Split out the checksum code from the support directory
WT-2600 Clean up test program #includes
WT-2602 LSM stress hangs with very large uncompressed pages
WT-2605 C tests focused on different schema types needed for join and other APIs
WT-2609 Incorrect "skips API_END call" error.
WT-2610 Investigate eviction hazard pointer check
WT-2611 wtperf and/or WT_CONFIG_PARSER do not allow escaped double quotes
WT-2612 The dist/s_prototypes script is creating a debugging file xxx.
WT-2613 Windows build failing with a C4100 error
WT-2615 Enabling checkpoints in test/format leads to reduced concurrency
WT-2616 In-memory deadlock getting size
WT-2617 Bug in pluggable file system example
WT-2621 WiredTiger fails to compile on MSVC 2013
WT-2622 Unit test failed in test_cursor_random.test_cursor_random.test_cursor_random_multiple_page_records
WT-2624 snprintf unsupported on MSVC 2013
WT-2626 snprintf errors with macro redefinition on MSVC2015
WT-2627 Coverity complaints
WT-2628 Reconciliation can return without unlocking the page lock
WT-2629 Introduction of ppc64le crc32c assembly file has made the stack executable
WT-2630 Rename WT_FSTREAM methods to avoid using C99 reserved names
WT-2632 Test format failures due to "checkpoints cannot be dropped when in-use"
WT-2637 The file-extension documentation doesn't cover not-supported cases
WT-2644 'wt load -r' (rename) fails with LSM
WT-2646 Split the lock_wait flag into two, adding a checkpoint_wait flag
WT-2651 Coverity 1355591 resource leak
WT-2652 Logging code improvements
WT-2653 The custom file-system example should show device configuration
WT-2656 Builds failing on GCC 4.7 builder
WT-2658 Only include PPC-specific files in PPC builds
WT-2659 csuite tests, assorted lint and cleanup.
WT-2660 Hang between eviction and connection close
WT-2661 Coverity failures: 1356050-1356053
WT-2662 For internal spell checking, strip out double quote literals, they confuse aspell
WT-2664 Change eviction so any eviction thread can find candidates
WT-2667 Enhance WiredTiger Evergreen testing
WT-2668 Create join statistics that are useful and are easy to understand
WT-2671 Dump more information about the file layout in verify debug mode
WT-2672 Handle system calls that don't set errno
WT-2673 Stop automatically increasing memory page max
WT-2674 Simplify metadata file check
WT-2676 Don't use key size in column store in-memory splits.
WT-2677 Fix JSON output so only printable ASCII is produced (seen on Solaris)
WT-2682 Add option to configure WiredTiger with strict compiler flags
WT-2683 WiredTiger no longer needs to return non-zero disk sizes
WT-2685 Hazard pointer failure from clear walk
WT-2686 Logging subsystem core dump
WT-2687 Test suite should verify the exit status of the wt utility
WT-2689 Use after free in WT_SESSION::open_cursor
WT-2691 Use wrappers for ctype functions to avoid sign extension errors
WT-2692 Fix race in file system example
WT-2696 Race condition on unclean shutdown may miss log records with large updates
WT-2698 Test/recovery hung
WT-2702 Under high thread load, WiredTiger exceeds cache size
WT-2704 Test/format hung on bengal
WT-2706 Race condition on log file switch can cause missing log records
WT-2707 dist/s_label enhancements, and error jump cleanups
WT-2709 Connection reconfigure segfault in __wt_conn_cache_pool_destroy
WT-2710 WT_FILE_HANDLE_INMEM no longer needs an off field
WT-2712 Coverity 1356928 and 1356929: ASSERT_SIDE_EFFECT
WT-2713 Document WT_PANIC so pluggable filesystems can panic.
WT-2714 Lint
WT-2715 random-abort test may write partial record at the end
WT-2720 Pull request tester not running Python suite
WT-2722 s_label or s_label_loop false positive
WT-2724 Eviction workers created on open exit immediately
WT-2763 Unit test test_intpack failing on OSX
Branch: v3.2
https://github.com/mongodb/mongo/commit/385faa6ca72ad599d92d3948f1a2033d908e0226

Author:

Message: Import wiredtiger-wiredtiger-2.8.0-592-g848e5f5.tar.gz from wiredtiger branch mongodb-3.2

ref: 8b7110b..848e5f5
for: 3.2.9

This commit replaces a number of previous backports with the original
change, so some of the tickets listed below may have been resolved in
previous versions already as a backport to the v3.2 branch. For example,
SERVER-24580 includes change a63e21b8 as a cherry-pick of 1f4aaa44, but
this commit replaces that change with 1f4aaa44 itself.

SERVER-23659 Provide useful message when wiredTigerJournalCompressor is changed
SERVER-23661 $sample takes disproportionately long time on newly created collection
SERVER-24306 40-second journaling stall from "log files prepared" to checkpoint
SERVER-24580 Improve performance when WiredTiger cache is full
WT-2103 Add incremental backup testing to format
WT-2223 Add stress testing for in-memory
WT-2268 JSON load incorrect with UNICODE input
WT-2319 Add statistics around fsync calls
WT-2325 Fix an incomplete comment
WT-2343 Assert we don't remove or rename when backup cursor is open
WT-2349 Add ability to open databases read-only
WT-2359 WiredTiger with Python will hang if a calloc failure occurs during __wt_connection_close
WT-2360 Allow disjunctions and combinations of operations in join cursors
WT-2408 Windows error translation layer
WT-2446 Estimate WT cache hit ratio
WT-2450 Salvage releases pages, then explicitly evicts them.
WT-2453 Throughput drop in wtperf evict Jenkins tests
WT-2479 Dump utility discards table config (JSON)
WT-2491 The dhandle close_lock isn't valuable at the moment
WT-2504 Should READONLY always read basecfg file?
WT-2505 Review clang analyzer warnings
WT-2508 Test programs should remove test directories on the "clean" target
WT-2514 Log path name is an empty string.
WT-2518 LSM checkpoint handle acquisition optimization
WT-2520 WT_SESSION::verify should not alter tables
WT-2526 Mixing and matching readonly and read/write handles
WT-2535 Extend test/format to test for transactions reading their writes
WT-2537 Cannot open DB written by WT2.6.1 with WT2.8.0 due to WT_NOTFOUND on recovery
WT-2539 Implement file streaming above pluggable filesystems
WT-2540 Separate stream and file handle methods
WT-2541 Add statistics for number of threads currently in read/write
WT-2542 Fixed-length column store reconciliation overwrites original values
WT-2544 Fix eviction statistics when clear is configured
WT-2546 Eviction server not help evict pages sometimes
WT-2547 Add 1-eviction-worker jobs to Jenkins
WT-2548 Cap the amount of data handed to raw compression.
WT-2549 joins using recno keys return no values
WT-2550 java ex_schema example fails
WT-2552 Public API for pluggable filesystems
WT-2553 Document in-memory configuration and WT_CACHE_FULL error return
WT-2554 Implement a framework for adding C test cases
WT-2556 Typo in the Java example code
WT-2557 format test program should discard log files after incremental backup
WT-2558 WT_PAGE structure reorganization
WT-2559 Windows segfault in logging code
WT-2560 Stuck trying to update oldest transaction ID
WT-2562 Reconfig02 test failing sometimes on PPC
WT-2565 Item 3573 on page at [write-check] is a corrupted cell
WT-2566 All lock operations should be barriers
WT-2567 Segfault in test/format log truncate
WT-2568 Java PackTest.java compilation error
WT-2569 Win_handle_read should always call GetLastError on error
WT-2570 Minor lint cleanups.
WT-2571 Join code cleanup
WT-2572 Don't select an in-memory format run if incompatible options configured
WT-2573 Free of stack-allocated WT_REF
WT-2574 Format doesn't free all allocated configure memory
WT-2576 Variable-length column-store out-of-order return
WT-2577 Core dump discarding non-existent addresses
WT-2579 In-memory configurations break debugging support
WT-2580 Potential SWIG naming conflict in Java
WT-2581 Assert multi->disk_image == NULL
WT-2582 Cache eviction server error: WT_RESTART
WT-2583 Incremental backup can prevent future recovery
WT-2584 Don't use periods in error messages
WT-2586 Remove ex_config.c until config cursors are supported
WT-2589 Check stats using WT_STAT_SET and 'clear' usage
WT-2592 Joins using non-recno key types not working
WT-2593 Disk full with pre-allocated log files
WT-2595 Fix compiler warning in packing tests
WT-2597 JSON dump needs LSM tests
WT-2598 In-memory FS needs fast lookup on file names
WT-2599 Split out the checksum code from the support directory
WT-2600 Clean up test program #includes
WT-2602 LSM stress hangs with very large uncompressed pages
WT-2605 C tests focused on different schema types needed for join and other APIs
WT-2609 Incorrect "skips API_END call" error.
WT-2610 Investigate eviction hazard pointer check
WT-2611 wtperf and/or WT_CONFIG_PARSER do not allow escaped double quotes
WT-2612 The dist/s_prototypes script is creating a debugging file xxx.
WT-2613 Windows build failing with a C4100 error
WT-2615 Enabling checkpoints in test/format leads to reduced concurrency
WT-2616 In-memory deadlock getting size
WT-2617 Bug in pluggable file system example
WT-2621 WiredTiger fails to compile on MSVC 2013
WT-2622 Unit test failed in test_cursor_random.test_cursor_random.test_cursor_random_multiple_page_records
WT-2624 snprintf unsupported on MSVC 2013
WT-2626 snprintf errors with macro redefinition on MSVC2015
WT-2627 Coverity complaints
WT-2628 Reconciliation can return without unlocking the page lock
WT-2629 Introduction of ppc64le crc32c assembly file has made the stack executable
WT-2630 Rename WT_FSTREAM methods to avoid using C99 reserved names
WT-2632 Test format failures due to "checkpoints cannot be dropped when in-use"
WT-2637 The file-extension documentation doesn't cover not-supported cases
WT-2644 'wt load -r' (rename) fails with LSM
WT-2646 Split the lock_wait flag into two, adding a checkpoint_wait flag
WT-2651 Coverity 1355591 resource leak
WT-2652 Logging code improvements
WT-2653 The custom file-system example should show device configuration
WT-2656 Builds failing on GCC 4.7 builder
WT-2658 Only include PPC-specific files in PPC builds
WT-2659 csuite tests, assorted lint and cleanup.
WT-2660 Hang between eviction and connection close
WT-2661 Coverity failures: 1356050-1356053
WT-2662 For internal spell checking, strip out double quote literals, they confuse aspell
WT-2664 Change eviction so any eviction thread can find candidates
WT-2667 Enhance WiredTiger Evergreen testing
WT-2668 Create join statistics that are useful and are easy to understand
WT-2671 Dump more information about the file layout in verify debug mode
WT-2672 Handle system calls that don't set errno
WT-2673 Stop automatically increasing memory page max
WT-2674 Simplify metadata file check
WT-2676 Don't use key size in column store in-memory splits.
WT-2677 Fix JSON output so only printable ASCII is produced (seen on Solaris)
WT-2682 Add option to configure WiredTiger with strict compiler flags
WT-2683 WiredTiger no longer needs to return non-zero disk sizes
WT-2685 Hazard pointer failure from clear walk
WT-2686 Logging subsystem core dump
WT-2687 Test suite should verify the exit status of the wt utility
WT-2689 Use after free in WT_SESSION::open_cursor
WT-2691 Use wrappers for ctype functions to avoid sign extension errors
WT-2692 Fix race in file system example
WT-2696 Race condition on unclean shutdown may miss log records with large updates
WT-2698 Test/recovery hung
WT-2702 Under high thread load, WiredTiger exceeds cache size
WT-2704 Test/format hung on bengal
WT-2706 Race condition on log file switch can cause missing log records
WT-2707 dist/s_label enhancements, and error jump cleanups
WT-2709 Connection reconfigure segfault in __wt_conn_cache_pool_destroy
WT-2710 WT_FILE_HANDLE_INMEM no longer needs an off field
WT-2712 Coverity 1356928 and 1356929: ASSERT_SIDE_EFFECT
WT-2713 Document WT_PANIC so pluggable filesystems can panic.
WT-2714 Lint
WT-2715 random-abort test may write partial record at the end
WT-2720 Pull request tester not running Python suite
WT-2722 s_label or s_label_loop false positive
WT-2724 Eviction workers created on open exit immediately
WT-2763 Unit test test_intpack failing on OSX
Branch: v3.2
https://github.com/mongodb/mongo/commit/385faa6ca72ad599d92d3948f1a2033d908e0226