Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.12, WT2.9.1, 3.5.1, 3.4.2
    • Labels:
      None
    • # Replies:
      7
    • Last comment by Customer:
      true

      Description

      This occurs on Mac OS/X (not yet test on other platforms).

      java org.junit.runner.JUnitCore com.wiredtiger.test.AsyncTest
      JUnit version 4.11
      .[thread 22787 also had an error]#

      # A fatal error has been detected by the Java Runtime Environment:
      #
      # SIGSEGV (0xb) at pc=0x000000012d3ef85f, pid=53369, tid=23043
      [thread 23299 also had an error]#

      # JRE version: Java(TM) SE Runtime Environment (8.0-b132) (build 1.8.0-b132)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode bsd-amd64 compressed oops)
      # Problematic frame:
      # C [libwiredtiger_java.0.dylib+0x385f] _wt_async_op_java_raw+0x1f
      #
      # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
      #
      # An error report file with more information is saved as:
      # /Users/dda/wt/git/tmp1/build_posix/lang/java/hs_err_pid53369.log
      #
      # If you would like to submit a bug report, please visit:
      # http://bugreport.sun.com/bugreport/crash.jsp
      # The crash happened outside the Java Virtual Machine in native code.
      # See problematic frame for where to report the bug.
      #
      ./SomeJunitTests: line 8: 53369 Abort trap: 6 java org.junit.runner.JUnitCore com.wiredtiger.test.AsyncTest

        Activity

        Hide
        donald.anderson Donald Anderson added a comment - - edited

        The problem is when the application calls a function that needs to unpack a key or value, here's a Java traceback at the point of failure:

        com.wiredtiger.db.WiredTigerException: Invalid argument
        at com.wiredtiger.db.wiredtigerJNI.AsyncOp_get_key_wrap(Native Method)
        at com.wiredtiger.db.AsyncOp.get_key_wrap(AsyncOp.java:576)
        at com.wiredtiger.db.AsyncOp.getKeyUnpacker(AsyncOp.java:526)
        at com.wiredtiger.db.AsyncOp.getKeyString(AsyncOp.java:366)
        at com.wiredtiger.test.AsyncTest.asyncTester(AsyncTest.java:306)
        at com.wiredtiger.test.AsyncTest.async01(AsyncTest.java:376)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        In the C code, the get_key function calls an internal (to the SWIG layer) _cursor_raw() function, when it faults, it is because the WT_ASYNC_OP->lang_private is NULL. (AsyncOp::_cursor_raw() is a native java function that uses lang_private reaches into the "java callback object", which has info about cursor raw-ness and lots of other stuff). Note also that the failure is timing related, it doesn't happen on every get_key call. Turns out that the SWIG layer callback function for async methods (which will call a Java method) sets lang_private to NULL after stashing its value. So when _cursor_raw() is called, lang_private may be NULL. When the test was written it worked, but when the _cursor_raw() code was added, it became subject to failure.

        A simple fix is in review to cache the value of _cursor_raw() when the Java AsyncOp object is created.

        Show
        donald.anderson Donald Anderson added a comment - - edited The problem is when the application calls a function that needs to unpack a key or value, here's a Java traceback at the point of failure: com.wiredtiger.db.WiredTigerException: Invalid argument at com.wiredtiger.db.wiredtigerJNI.AsyncOp_get_key_wrap(Native Method) at com.wiredtiger.db.AsyncOp.get_key_wrap(AsyncOp.java:576) at com.wiredtiger.db.AsyncOp.getKeyUnpacker(AsyncOp.java:526) at com.wiredtiger.db.AsyncOp.getKeyString(AsyncOp.java:366) at com.wiredtiger.test.AsyncTest.asyncTester(AsyncTest.java:306) at com.wiredtiger.test.AsyncTest.async01(AsyncTest.java:376) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) In the C code, the get_key function calls an internal (to the SWIG layer) _cursor_raw() function, when it faults, it is because the WT_ASYNC_OP->lang_private is NULL. (AsyncOp::_cursor_raw() is a native java function that uses lang_private reaches into the "java callback object", which has info about cursor raw-ness and lots of other stuff). Note also that the failure is timing related, it doesn't happen on every get_key call. Turns out that the SWIG layer callback function for async methods (which will call a Java method) sets lang_private to NULL after stashing its value. So when _cursor_raw() is called, lang_private may be NULL. When the test was written it worked, but when the _cursor_raw() code was added, it became subject to failure. A simple fix is in review to cache the value of _cursor_raw() when the Java AsyncOp object is created.
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

        Message: WT-3055 When an AsyncOp is created, cache the whether the cursor is "raw". (#3182)

        Later, when an application is fetching keys/values from the AsyncOp,
        the JAVA_CALLBACK object may be detached from the (java) AsyncOp, which makes
        finding out information about the cursor impossible.
        Branch: develop
        https://github.com/wiredtiger/wiredtiger/commit/c68e35c4c8aca43ebfebaf1ed8a68bef71ccb4cc

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'} Message: WT-3055 When an AsyncOp is created, cache the whether the cursor is "raw". (#3182) Later, when an application is fetching keys/values from the AsyncOp, the JAVA_CALLBACK object may be detached from the (java) AsyncOp, which makes finding out information about the cursor impossible. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/c68e35c4c8aca43ebfebaf1ed8a68bef71ccb4cc
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

        Message: Import wiredtiger: 1b6c815a3fd34f14c20d5cd627155799d1de535c from branch mongodb-3.6

        ref: ca6eee06ff..1b6c815a3f
        for: 3.5.1

        WT-2336 Add a test validating schema operations via file system call monitoring
        WT-2670 Add option to configure read-ahead per table and change default behavior
        WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
        WT-2969 Fix a bug that could cause snapshot corruption during compaction
        WT-3014 Add GCC/clang support for ELF symbol visibility.
        WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
        WT-3025 fix error path in log_force_sync
        WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
        WT-3030 Test failure indicating invalid key order during traversal
        WT-3034 Add support for single-writer named snapshots.
        WT-3037 Fix some outdated comments in logging
        WT-3048 WiredTiger maximum size warning uses the wrong format.
        WT-3051 Remove external __wt_hex symbol.
        WT-3052 Improve search if an index hint is wrong
        WT-3053 Review Python and Java calls to internal WiredTiger functions
        WT-3054 Java PackTest, PackTest03 do not compile
        WT-3055 Java AsyncTest faults
        WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
        WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
        Branch: master
        https://github.com/mongodb/mongo/commit/21a6f07d859c132154166bd3d83bbed238d5d719

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: 1b6c815a3fd34f14c20d5cd627155799d1de535c from branch mongodb-3.6 ref: ca6eee06ff..1b6c815a3f for: 3.5.1 WT-2336 Add a test validating schema operations via file system call monitoring WT-2670 Add option to configure read-ahead per table and change default behavior WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Fix a bug that could cause snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3064 minor tree cleanups: .gitignore, NEWS misspelling Branch: master https://github.com/mongodb/mongo/commit/21a6f07d859c132154166bd3d83bbed238d5d719
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

        Message: WT-3055 When an AsyncOp is created, cache the whether the cursor is "raw". (#3182)

        Later, when an application is fetching keys/values from the AsyncOp,
        the JAVA_CALLBACK object may be detached from the (java) AsyncOp, which makes
        finding out information about the cursor impossible.
        Branch: mongodb-3.4
        https://github.com/wiredtiger/wiredtiger/commit/c68e35c4c8aca43ebfebaf1ed8a68bef71ccb4cc

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'} Message: WT-3055 When an AsyncOp is created, cache the whether the cursor is "raw". (#3182) Later, when an application is fetching keys/values from the AsyncOp, the JAVA_CALLBACK object may be detached from the (java) AsyncOp, which makes finding out information about the cursor impossible. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/c68e35c4c8aca43ebfebaf1ed8a68bef71ccb4cc
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

        Message: WT-3055 When an AsyncOp is created, cache the whether the cursor is "raw". (#3182)

        Later, when an application is fetching keys/values from the AsyncOp,
        the JAVA_CALLBACK object may be detached from the (java) AsyncOp, which makes
        finding out information about the cursor impossible.
        Branch: mongodb-3.2
        https://github.com/wiredtiger/wiredtiger/commit/c68e35c4c8aca43ebfebaf1ed8a68bef71ccb4cc

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'} Message: WT-3055 When an AsyncOp is created, cache the whether the cursor is "raw". (#3182) Later, when an application is fetching keys/values from the AsyncOp, the JAVA_CALLBACK object may be detached from the (java) AsyncOp, which makes finding out information about the cursor impossible. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/c68e35c4c8aca43ebfebaf1ed8a68bef71ccb4cc
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

        Message: Import wiredtiger: d48181f6f4db08761ed7b80b0332908b272ad0d0 from branch mongodb-3.2

        ref: 040e3d6f76..d48181f6f4
        for: 3.2.12

        SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
        WT-2336 Add a test validating schema operations via file system call monitoring
        WT-2402 Misaligned structure accesses lead to undefined behavior
        WT-2670 Inefficient I/O when read full DB (poor readahead)
        WT-283 Add a way to change persistent object settings
        WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
        WT-2969 Possible snapshot corruption during compaction
        WT-3014 Add GCC/clang support for ELF symbol visibility.
        WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
        WT-3025 fix error path in log_force_sync
        WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
        WT-3030 Test failure indicating invalid key order during traversal
        WT-3034 Add support for single-writer named snapshots.
        WT-3037 Fix some outdated comments in logging
        WT-3048 WiredTiger maximum size warning uses the wrong format.
        WT-3051 Remove external __wt_hex symbol.
        WT-3052 Improve search if an index hint is wrong
        WT-3053 Review Python and Java calls to internal WiredTiger functions
        WT-3054 Java PackTest, PackTest03 do not compile
        WT-3055 Java AsyncTest faults
        WT-3056 For cursors with projections, keys should be allowed
        WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
        WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
        WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
        WT-3066 lint
        WT-3068 Copy wtperf artifacts when running Jenkins tests
        WT-3069 Fix build failures in LevelDB APIs
        WT-3070 Fix search_near() for index cursor
        WT-3071 Java: fix build with -Werror=sign-conversion
        WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
        WT-3078 Fix a hang in the reconfiguration test.
        WT-3084 Fix Coverity resource leak complaint.
        Branch: v3.2
        https://github.com/mongodb/mongo/commit/52b68fa86ea43e909ad42c901d0579bced6b205f

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: Import wiredtiger: d48181f6f4db08761ed7b80b0332908b272ad0d0 from branch mongodb-3.2 ref: 040e3d6f76..d48181f6f4 for: 3.2.12 SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers WT-2336 Add a test validating schema operations via file system call monitoring WT-2402 Misaligned structure accesses lead to undefined behavior WT-2670 Inefficient I/O when read full DB (poor readahead) WT-283 Add a way to change persistent object settings WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Possible snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3056 For cursors with projections, keys should be allowed WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64 WT-3064 minor tree cleanups: .gitignore, NEWS misspelling WT-3066 lint WT-3068 Copy wtperf artifacts when running Jenkins tests WT-3069 Fix build failures in LevelDB APIs WT-3070 Fix search_near() for index cursor WT-3071 Java: fix build with -Werror=sign-conversion WT-3075 Document and enforce that WiredTiger now depends on Python 2.7 WT-3078 Fix a hang in the reconfiguration test. WT-3084 Fix Coverity resource leak complaint. Branch: v3.2 https://github.com/mongodb/mongo/commit/52b68fa86ea43e909ad42c901d0579bced6b205f
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

        Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.4

        ref: ca6eee06ff..8d23249433
        for: 3.4.2

        SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
        WT-2336 Add a test validating schema operations via file system call monitoring
        WT-2402 Misaligned structure accesses lead to undefined behavior
        WT-2670 Inefficient I/O when read full DB (poor readahead)
        WT-283 Add a way to change persistent object settings
        WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
        WT-2969 Possible snapshot corruption during compaction
        WT-3014 Add GCC/clang support for ELF symbol visibility.
        WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
        WT-3025 fix error path in log_force_sync
        WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
        WT-3030 Test failure indicating invalid key order during traversal
        WT-3034 Add support for single-writer named snapshots.
        WT-3037 Fix some outdated comments in logging
        WT-3048 WiredTiger maximum size warning uses the wrong format.
        WT-3051 Remove external __wt_hex symbol.
        WT-3052 Improve search if an index hint is wrong
        WT-3053 Review Python and Java calls to internal WiredTiger functions
        WT-3054 Java PackTest, PackTest03 do not compile
        WT-3055 Java AsyncTest faults
        WT-3056 For cursors with projections, keys should be allowed
        WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
        WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
        WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
        WT-3066 lint
        WT-3068 Copy wtperf artifacts when running Jenkins tests
        WT-3069 Fix build failures in LevelDB APIs
        WT-3070 Fix search_near() for index cursor
        WT-3071 Java: fix build with -Werror=sign-conversion
        WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
        WT-3078 Fix a hang in the reconfiguration test.
        WT-3084 Fix Coverity resource leak complaint.
        Branch: v3.4
        https://github.com/mongodb/mongo/commit/d2c64ac8c526b70eadeb859ec41370a5f03a64aa

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.4 ref: ca6eee06ff..8d23249433 for: 3.4.2 SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers WT-2336 Add a test validating schema operations via file system call monitoring WT-2402 Misaligned structure accesses lead to undefined behavior WT-2670 Inefficient I/O when read full DB (poor readahead) WT-283 Add a way to change persistent object settings WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Possible snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3056 For cursors with projections, keys should be allowed WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64 WT-3064 minor tree cleanups: .gitignore, NEWS misspelling WT-3066 lint WT-3068 Copy wtperf artifacts when running Jenkins tests WT-3069 Fix build failures in LevelDB APIs WT-3070 Fix search_near() for index cursor WT-3071 Java: fix build with -Werror=sign-conversion WT-3075 Document and enforce that WiredTiger now depends on Python 2.7 WT-3078 Fix a hang in the reconfiguration test. WT-3084 Fix Coverity resource leak complaint. Branch: v3.4 https://github.com/mongodb/mongo/commit/d2c64ac8c526b70eadeb859ec41370a5f03a64aa

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Days since reply:
              11 weeks, 2 days ago
              Date of 1st Reply: