Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3021

Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.12, WT2.9.1, 3.5.1, 3.4.2
    • Labels:
      None
    • # Replies:
      7
    • Last comment by Customer:
      true
    • Sprint:
      Storage 2016-12-12

      Description

      Reported on wiredtiger-users group by user "Sand Stone":

      I am playing with the log cursor sample, ex_log.java. It seems the
      example is broken with mongodb-3.4 and develop at least.

      After fixing the minor sample code issue, there are two main issues:
      1. the LSN offset type is changed from long to int.
      2. the cursor.search seems always return -31803

      I've confirmed it's broken:

      com.wiredtiger.examples.ex_log
      Exception: com.wiredtiger.db.WiredTigerPackingException: Format mismatch. Wanted: q, got: I
      com.wiredtiger.db.WiredTigerPackingException: Format mismatch. Wanted: q, got: I
      at com.wiredtiger.db.PackFormatInputStream.checkType(PackFormatInputStream.java:123)
      at com.wiredtiger.db.PackInputStream.getLong(PackInputStream.java:221)
      at com.wiredtiger.db.Cursor.getKeyLong(Cursor.java:322)
      at com.wiredtiger.examples.ex_log.simple_walk_log(ex_log.java:141)
      at com.wiredtiger.examples.ex_log.logExample(ex_log.java:363)
      at com.wiredtiger.examples.ex_log.main(ex_log.java:374)

        Activity

        Hide
        donald.anderson Donald Anderson added a comment -

        The Java example has some flaws that prevent it from running. The C logging example (on which the Java example is based), gets raw records from log cursors and inserts them into a table using a raw mode cursor, later reading them using regular (cooked?) mode. This exposed two additional errors. One is that Java did not handle writing to a raw mode cursor correctly. Java turns on raw mode when interfacing to C, but it must also handle the case when that application wants raw mode to write records as byte arrays. The other was that log cursors did not handle raw mode correctly during the search API.

        There's a pull request in review that should fix all these problems, and allow Java to fully use log cursors.

        Show
        donald.anderson Donald Anderson added a comment - The Java example has some flaws that prevent it from running. The C logging example (on which the Java example is based), gets raw records from log cursors and inserts them into a table using a raw mode cursor, later reading them using regular (cooked?) mode. This exposed two additional errors. One is that Java did not handle writing to a raw mode cursor correctly. Java turns on raw mode when interfacing to C, but it must also handle the case when that application wants raw mode to write records as byte arrays. The other was that log cursors did not handle raw mode correctly during the search API. There's a pull request in review that should fix all these problems, and allow Java to fully use log cursors.
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

        Message: WT-3021 fixes for java log example, raw mode in java, and raw mode in log cursors (#3150)

        • log cursors performing search must internally get keys; raw mode must be turned off temporarily to accomplish that. Use this same technique to clean up how keys/values are set.
        • In java, when a raw mode cursor is created, allow setting key/value only via byte array.
        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'} Message: WT-3021 fixes for java log example, raw mode in java, and raw mode in log cursors (#3150) log cursors performing search must internally get keys; raw mode must be turned off temporarily to accomplish that. Use this same technique to clean up how keys/values are set. In java, when a raw mode cursor is created, allow setting key/value only via byte array. Synchronize java log example with the C version, including changing LSN from long to int and adding a count check to the simple log walk. Fix an error with how LSN file/offset was saved for a later search. The 'copy' table can now use S formats, since there is a way to insert them as raw records obtained from the log. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/938f6252f3dfa3cfca86fa9ebf94a4e529c1e035
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

        Message: Import wiredtiger: 1b6c815a3fd34f14c20d5cd627155799d1de535c from branch mongodb-3.6

        ref: ca6eee06ff..1b6c815a3f
        for: 3.5.1

        WT-2336 Add a test validating schema operations via file system call monitoring
        WT-2670 Add option to configure read-ahead per table and change default behavior
        WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
        WT-2969 Fix a bug that could cause snapshot corruption during compaction
        WT-3014 Add GCC/clang support for ELF symbol visibility.
        WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
        WT-3025 fix error path in log_force_sync
        WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
        WT-3030 Test failure indicating invalid key order during traversal
        WT-3034 Add support for single-writer named snapshots.
        WT-3037 Fix some outdated comments in logging
        WT-3048 WiredTiger maximum size warning uses the wrong format.
        WT-3051 Remove external __wt_hex symbol.
        WT-3052 Improve search if an index hint is wrong
        WT-3053 Review Python and Java calls to internal WiredTiger functions
        WT-3054 Java PackTest, PackTest03 do not compile
        WT-3055 Java AsyncTest faults
        WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
        WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
        Branch: master
        https://github.com/mongodb/mongo/commit/21a6f07d859c132154166bd3d83bbed238d5d719

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: 1b6c815a3fd34f14c20d5cd627155799d1de535c from branch mongodb-3.6 ref: ca6eee06ff..1b6c815a3f for: 3.5.1 WT-2336 Add a test validating schema operations via file system call monitoring WT-2670 Add option to configure read-ahead per table and change default behavior WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Fix a bug that could cause snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3064 minor tree cleanups: .gitignore, NEWS misspelling Branch: master https://github.com/mongodb/mongo/commit/21a6f07d859c132154166bd3d83bbed238d5d719
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

        Message: WT-3021 fixes for java log example, raw mode in java, and raw mode in log cursors (#3150)

        • log cursors performing search must internally get keys; raw mode must be turned off temporarily to accomplish that. Use this same technique to clean up how keys/values are set.
        • In java, when a raw mode cursor is created, allow setting key/value only via byte array.
        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'} Message: WT-3021 fixes for java log example, raw mode in java, and raw mode in log cursors (#3150) log cursors performing search must internally get keys; raw mode must be turned off temporarily to accomplish that. Use this same technique to clean up how keys/values are set. In java, when a raw mode cursor is created, allow setting key/value only via byte array. Synchronize java log example with the C version, including changing LSN from long to int and adding a count check to the simple log walk. Fix an error with how LSN file/offset was saved for a later search. The 'copy' table can now use S formats, since there is a way to insert them as raw records obtained from the log. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/938f6252f3dfa3cfca86fa9ebf94a4e529c1e035
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

        Message: WT-3021 fixes for java log example, raw mode in java, and raw mode in log cursors (#3150)

        • log cursors performing search must internally get keys; raw mode must be turned off temporarily to accomplish that. Use this same technique to clean up how keys/values are set.
        • In java, when a raw mode cursor is created, allow setting key/value only via byte array.
        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'} Message: WT-3021 fixes for java log example, raw mode in java, and raw mode in log cursors (#3150) log cursors performing search must internally get keys; raw mode must be turned off temporarily to accomplish that. Use this same technique to clean up how keys/values are set. In java, when a raw mode cursor is created, allow setting key/value only via byte array. Synchronize java log example with the C version, including changing LSN from long to int and adding a count check to the simple log walk. Fix an error with how LSN file/offset was saved for a later search. The 'copy' table can now use S formats, since there is a way to insert them as raw records obtained from the log. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/938f6252f3dfa3cfca86fa9ebf94a4e529c1e035
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

        Message: Import wiredtiger: d48181f6f4db08761ed7b80b0332908b272ad0d0 from branch mongodb-3.2

        ref: 040e3d6f76..d48181f6f4
        for: 3.2.12

        SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
        WT-2336 Add a test validating schema operations via file system call monitoring
        WT-2402 Misaligned structure accesses lead to undefined behavior
        WT-2670 Inefficient I/O when read full DB (poor readahead)
        WT-283 Add a way to change persistent object settings
        WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
        WT-2969 Possible snapshot corruption during compaction
        WT-3014 Add GCC/clang support for ELF symbol visibility.
        WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
        WT-3025 fix error path in log_force_sync
        WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
        WT-3030 Test failure indicating invalid key order during traversal
        WT-3034 Add support for single-writer named snapshots.
        WT-3037 Fix some outdated comments in logging
        WT-3048 WiredTiger maximum size warning uses the wrong format.
        WT-3051 Remove external __wt_hex symbol.
        WT-3052 Improve search if an index hint is wrong
        WT-3053 Review Python and Java calls to internal WiredTiger functions
        WT-3054 Java PackTest, PackTest03 do not compile
        WT-3055 Java AsyncTest faults
        WT-3056 For cursors with projections, keys should be allowed
        WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
        WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
        WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
        WT-3066 lint
        WT-3068 Copy wtperf artifacts when running Jenkins tests
        WT-3069 Fix build failures in LevelDB APIs
        WT-3070 Fix search_near() for index cursor
        WT-3071 Java: fix build with -Werror=sign-conversion
        WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
        WT-3078 Fix a hang in the reconfiguration test.
        WT-3084 Fix Coverity resource leak complaint.
        Branch: v3.2
        https://github.com/mongodb/mongo/commit/52b68fa86ea43e909ad42c901d0579bced6b205f

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: Import wiredtiger: d48181f6f4db08761ed7b80b0332908b272ad0d0 from branch mongodb-3.2 ref: 040e3d6f76..d48181f6f4 for: 3.2.12 SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers WT-2336 Add a test validating schema operations via file system call monitoring WT-2402 Misaligned structure accesses lead to undefined behavior WT-2670 Inefficient I/O when read full DB (poor readahead) WT-283 Add a way to change persistent object settings WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Possible snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3056 For cursors with projections, keys should be allowed WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64 WT-3064 minor tree cleanups: .gitignore, NEWS misspelling WT-3066 lint WT-3068 Copy wtperf artifacts when running Jenkins tests WT-3069 Fix build failures in LevelDB APIs WT-3070 Fix search_near() for index cursor WT-3071 Java: fix build with -Werror=sign-conversion WT-3075 Document and enforce that WiredTiger now depends on Python 2.7 WT-3078 Fix a hang in the reconfiguration test. WT-3084 Fix Coverity resource leak complaint. Branch: v3.2 https://github.com/mongodb/mongo/commit/52b68fa86ea43e909ad42c901d0579bced6b205f
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

        Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.4

        ref: ca6eee06ff..8d23249433
        for: 3.4.2

        SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
        WT-2336 Add a test validating schema operations via file system call monitoring
        WT-2402 Misaligned structure accesses lead to undefined behavior
        WT-2670 Inefficient I/O when read full DB (poor readahead)
        WT-283 Add a way to change persistent object settings
        WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
        WT-2969 Possible snapshot corruption during compaction
        WT-3014 Add GCC/clang support for ELF symbol visibility.
        WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
        WT-3025 fix error path in log_force_sync
        WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
        WT-3030 Test failure indicating invalid key order during traversal
        WT-3034 Add support for single-writer named snapshots.
        WT-3037 Fix some outdated comments in logging
        WT-3048 WiredTiger maximum size warning uses the wrong format.
        WT-3051 Remove external __wt_hex symbol.
        WT-3052 Improve search if an index hint is wrong
        WT-3053 Review Python and Java calls to internal WiredTiger functions
        WT-3054 Java PackTest, PackTest03 do not compile
        WT-3055 Java AsyncTest faults
        WT-3056 For cursors with projections, keys should be allowed
        WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
        WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
        WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
        WT-3066 lint
        WT-3068 Copy wtperf artifacts when running Jenkins tests
        WT-3069 Fix build failures in LevelDB APIs
        WT-3070 Fix search_near() for index cursor
        WT-3071 Java: fix build with -Werror=sign-conversion
        WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
        WT-3078 Fix a hang in the reconfiguration test.
        WT-3084 Fix Coverity resource leak complaint.
        Branch: v3.4
        https://github.com/mongodb/mongo/commit/d2c64ac8c526b70eadeb859ec41370a5f03a64aa

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.4 ref: ca6eee06ff..8d23249433 for: 3.4.2 SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers WT-2336 Add a test validating schema operations via file system call monitoring WT-2402 Misaligned structure accesses lead to undefined behavior WT-2670 Inefficient I/O when read full DB (poor readahead) WT-283 Add a way to change persistent object settings WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Possible snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3056 For cursors with projections, keys should be allowed WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64 WT-3064 minor tree cleanups: .gitignore, NEWS misspelling WT-3066 lint WT-3068 Copy wtperf artifacts when running Jenkins tests WT-3069 Fix build failures in LevelDB APIs WT-3070 Fix search_near() for index cursor WT-3071 Java: fix build with -Werror=sign-conversion WT-3075 Document and enforce that WiredTiger now depends on Python 2.7 WT-3078 Fix a hang in the reconfiguration test. WT-3084 Fix Coverity resource leak complaint. Branch: v3.4 https://github.com/mongodb/mongo/commit/d2c64ac8c526b70eadeb859ec41370a5f03a64aa

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Days since reply:
              11 weeks, 2 days ago
              Date of 1st Reply:

                Agile