Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2235

wt printlog option without unicode

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.8.0
    • Labels:
      None
    • # Replies:
      9
    • Last comment by Customer:
      true

      Description

      wt printlog prints out characters whenever it can. Sometimes that makes it difficult to look at the contents.

        Activity

        Hide
        sue.loverso Sue LoVerso added a comment -

        The keys written are (1, number) where it is 1 << 32 | number. The unfortunate thing is that JSON outputs a printable character whenever it can. So, for 1,427260, I see this key in printlog:
        "key": "\u00E5\u0001\u0000\u0006\u0084\u00FC",

        0x684FC == 427260. When I look at a different value, 427570, I see this key in printlog:
        key": "\u00E5\u0001\u0000\u0006\u00862"

        It looks like 0x6862, but what it really is (I only know this because I know approximately what values should be there and other keys nearby) is 0x686"2" or 0x68632 == 427570.

        It is kind of sad and unfortunate that it is printed that way. In particular, it makes it hard to view and harder to search consistently.

        Show
        sue.loverso Sue LoVerso added a comment - The keys written are (1, number) where it is 1 << 32 | number. The unfortunate thing is that JSON outputs a printable character whenever it can. So, for 1,427260, I see this key in printlog: "key": "\u00E5\u0001\u0000\u0006\u0084\u00FC", 0x684FC == 427260. When I look at a different value, 427570, I see this key in printlog: key": "\u00E5\u0001\u0000\u0006\u00862" It looks like 0x6862, but what it really is (I only know this because I know approximately what values should be there and other keys nearby) is 0x686"2" or 0x68632 == 427570. It is kind of sad and unfortunate that it is printed that way. In particular, it makes it hard to view and harder to search consistently.
        Hide
        donald.anderson Donald Anderson added a comment - - edited

        Sue LoVerso, Two approaches I can think of:
        1) have a wt printlog option that forces the Unicode escape for everything. For the example you gave:
        "key: "\u00E5\u0001\u0000\u0006\u0086\u0062"
        "value": "Hello"

        2) have a wt printlog option that emits hex in addition to the current way.
        "key: "\u00E5\u0001\u0000\u0006\u00862"
        "key-hex: "E50100068662"
        "value": "Hello"
        "value-hex": "48656C6C6F"

        I find option two much more readable and usable (you get the printable stuff when you need it, and the hex stuff when you need that).

        Show
        donald.anderson Donald Anderson added a comment - - edited Sue LoVerso , Two approaches I can think of: 1) have a wt printlog option that forces the Unicode escape for everything. For the example you gave: "key: "\u00E5\u0001\u0000\u0006\u0086\u0062" "value": "Hello" 2) have a wt printlog option that emits hex in addition to the current way. "key: "\u00E5\u0001\u0000\u0006\u00862" "key-hex: "E50100068662" "value": "Hello" "value-hex": "48656C6C6F" I find option two much more readable and usable (you get the printable stuff when you need it, and the hex stuff when you need that).
        Hide
        sue.loverso Sue LoVerso added a comment -

        I think option two will be more usable because then it can be predictably searched. By the time we're printing the contents of the log records, we're searching through it.

        Show
        sue.loverso Sue LoVerso added a comment - I think option two will be more usable because then it can be predictably searched. By the time we're printing the contents of the log records, we're searching through it.
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@ddanderson.com'}

        Message: WT-2235. Changes to wt printlog to add -x option and remove -p.

        wt printlog -x produces extra "-hex" fields when appropriate. For example,
        when printing a "row_put" entry, in addition to "key" and "value" (printed
        as JSON strings with Unicode escapes), there will also be "key-hex" and
        "key-value" entries in the JSON dictionary, containing a hex dump for each
        of these fields.

        For consistency with existing WT outputs, all hex output is in lower case.
        One visible change is that our Unicode escapes in JSON, also used for
        'wt dump j', now produce hex characters in lower case ('a''f'). Before,
        'wt dump j' produced upper case hex (using 'A''F').
        Branch: develop
        https://github.com/wiredtiger/wiredtiger/commit/c19c1a7d5d52d35ab6c7959a6738630fe77a4fa5

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@ddanderson.com'} Message: WT-2235 . Changes to wt printlog to add -x option and remove -p. wt printlog -x produces extra "-hex" fields when appropriate. For example, when printing a "row_put" entry, in addition to "key" and "value" (printed as JSON strings with Unicode escapes), there will also be "key-hex" and "key-value" entries in the JSON dictionary, containing a hex dump for each of these fields. For consistency with existing WT outputs, all hex output is in lower case. One visible change is that our Unicode escapes in JSON, also used for 'wt dump j', now produce hex characters in lower case ('a' 'f'). Before, 'wt dump j' produced upper case hex (using 'A' 'F'). Branch: develop https://github.com/wiredtiger/wiredtiger/commit/c19c1a7d5d52d35ab6c7959a6738630fe77a4fa5
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@ddanderson.com'}

        Message: WT-2235. Fix doc synopses for 'wt' to use 'R', not 'r', as the recovery flag.
        Branch: develop
        https://github.com/wiredtiger/wiredtiger/commit/faa5c92e1098b286ce6b0a1b7f3f49e424d7fb15

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@ddanderson.com'} Message: WT-2235 . Fix doc synopses for 'wt' to use 'R', not 'r', as the recovery flag. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/faa5c92e1098b286ce6b0a1b7f3f49e424d7fb15
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@ddanderson.com'}

        Message: WT-2235. Share an array used for converted binary to hex.
        Branch: develop
        https://github.com/wiredtiger/wiredtiger/commit/4a225dd68a667b87162e61f0c20b183522419fa0

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@ddanderson.com'} Message: WT-2235 . Share an array used for converted binary to hex. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/4a225dd68a667b87162e61f0c20b183522419fa0
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'sueloverso', u'name': u'sueloverso', u'email': u'sue@mongodb.com'}

        Message: Merge pull request #2371 from wiredtiger/wt-2235-printlog-hex

        WT-2235. Changes to wt printlog to add -x option and remove -p.
        Branch: develop
        https://github.com/wiredtiger/wiredtiger/commit/8d4e75d1d00fe44268373fee49ce970c4356e714

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'sueloverso', u'email': u'sue@mongodb.com'} Message: Merge pull request #2371 from wiredtiger/wt-2235-printlog-hex WT-2235 . Changes to wt printlog to add -x option and remove -p. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/8d4e75d1d00fe44268373fee49ce970c4356e714
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

        Message: Import wiredtiger-wiredtiger-2.7.0-269-g44463c5.tar.gz from wiredtiger branch mongodb-3.4

        ref: 3c2ad56..44463c5

        SERVER-21833 Compact does not release space to the system with WiredTiger
        WT-2060 Simplify aggregation of statistics
        WT-2099 Seeing memory underflow messages
        WT-2113 truncate01 sometimes fails
        WT-2177 Add a per-thread seed to random number generator
        WT-2198 bulk load and column store appends
        WT-2231 pinned page cursor searches could check parent keys
        WT-2235 wt printlog option without unicode
        WT-2245 WTPERF Truncate has no ability to catch up when it falls behind
        WT-2246 column-store append searches the leaf page; the maximum record number fails CRUD operations
        WT-2256 WTPERFs throttle option fires in bursts
        WT-2257 wtperf doesn't handle overriding workload config
        WT-2259 __wt_evict_file_exclusive_on() should clear WT_BTREE_NO_EVICTION on error
        WT-2260 Workloads evict internal pages unexpectedly
        WT-2262 Random sampling is skewed by tree shape
        WT-2265 Wiredtiger related change in ppc64le specific code block in gcc.h
        WT-2266 Add wtperf config to set if perf thresholds are fatal
        WT-2269 wtperf should dump its config everytime it runs
        WT-2272 Stress test assertion in the sweep server
        WT-2275 broken DB after application crash
        WT-2276 tool to decode checkpoint addr
        WT-2277 Remove WT check against big-endian systems
        WT-2279 Define WT_PAUSE(), WT_FULL_BARRIER(), etc when s390x is defined
        WT-2281 wtperf smoke.sh fails on ppc64le
        WT-2282 error in wt_txn_update_oldest verbose message test
        WT-2283 retry in txn_update_oldest results in a hang
        WT-2285 configure should set BUFFER_ALIGNMENT_DEFAULT to 4kb on linux
        WT-2289 failure in fast key check
        WT-2290 WT_SESSION.compact could be more effective.
        WT-2291 Random cursor walk inefficient in skip list only trees
        WT-2297 Fix off-by-one error in Huffman config file parsing
        WT-2299 upper-level WiredTiger code is reaching into the block manager
        WT-2301 Add reading a range to wtperf
        WT-2303 Build warning in wtperf
        WT-2304 wtperf crash dumping config
        WT-2307 Internal page splits can corrupt cursor iteration
        WT-2311 Support Sparc
        Branch: master
        https://github.com/mongodb/mongo/commit/d845b75e5f0837f801bdf371babd985308a1ad80

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger-wiredtiger-2.7.0-269-g44463c5.tar.gz from wiredtiger branch mongodb-3.4 ref: 3c2ad56..44463c5 SERVER-21833 Compact does not release space to the system with WiredTiger WT-2060 Simplify aggregation of statistics WT-2099 Seeing memory underflow messages WT-2113 truncate01 sometimes fails WT-2177 Add a per-thread seed to random number generator WT-2198 bulk load and column store appends WT-2231 pinned page cursor searches could check parent keys WT-2235 wt printlog option without unicode WT-2245 WTPERF Truncate has no ability to catch up when it falls behind WT-2246 column-store append searches the leaf page; the maximum record number fails CRUD operations WT-2256 WTPERFs throttle option fires in bursts WT-2257 wtperf doesn't handle overriding workload config WT-2259 __wt_evict_file_exclusive_on() should clear WT_BTREE_NO_EVICTION on error WT-2260 Workloads evict internal pages unexpectedly WT-2262 Random sampling is skewed by tree shape WT-2265 Wiredtiger related change in ppc64le specific code block in gcc.h WT-2266 Add wtperf config to set if perf thresholds are fatal WT-2269 wtperf should dump its config everytime it runs WT-2272 Stress test assertion in the sweep server WT-2275 broken DB after application crash WT-2276 tool to decode checkpoint addr WT-2277 Remove WT check against big-endian systems WT-2279 Define WT_PAUSE(), WT_FULL_BARRIER(), etc when s390x is defined WT-2281 wtperf smoke.sh fails on ppc64le WT-2282 error in wt_txn_update_oldest verbose message test WT-2283 retry in txn_update_oldest results in a hang WT-2285 configure should set BUFFER_ALIGNMENT_DEFAULT to 4kb on linux WT-2289 failure in fast key check WT-2290 WT_SESSION.compact could be more effective. WT-2291 Random cursor walk inefficient in skip list only trees WT-2297 Fix off-by-one error in Huffman config file parsing WT-2299 upper-level WiredTiger code is reaching into the block manager WT-2301 Add reading a range to wtperf WT-2303 Build warning in wtperf WT-2304 wtperf crash dumping config WT-2307 Internal page splits can corrupt cursor iteration WT-2311 Support Sparc Branch: master https://github.com/mongodb/mongo/commit/d845b75e5f0837f801bdf371babd985308a1ad80
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

        Message: Import wiredtiger-wiredtiger-2.7.0-559-g07966a4.tar.gz from wiredtiger branch mongodb-3.2

        ref: 3c2ad56..07966a4

        WT-1517 schema format edge cases
        WT-1801 Add a directory sync after rollback of a WT_SESSION::rename operation
        WT-2060 Simplify aggregation of statistics
        WT-2073 metadata cleanups
        WT-2099 Seeing memory underflow messages
        WT-2113 truncate01 sometimes fails
        WT-2142 Connection cleanup in Python tests
        WT-2177 Add an optional per-thread seed to random number generator
        WT-2198 bulk load and column store appends
        WT-2216 simplify row-store search loop slightly
        WT-2225 New split code performance impact
        WT-2231 pinned page cursor searches could check parent keys
        WT-2235 wt printlog option without unicode
        WT-2242 WiredTiger treats dead trees the same as other trees in eviction
        WT-2244 Trigger in-memory splits sooner
        WT-2245 WTPERF Truncate has no ability to catch up when it falls behind
        WT-2246 column-store append searches the leaf page; the maximum record number fails CRUD operations
        WT-2247 variable-length column-store in-memory page splits
        WT-2256 WTPERFs throttle option fires in bursts
        WT-2257 wtperf doesn't handle overriding workload config
        WT-2258 WiredTiger preloads pages even when direct-IO is configured.
        WT-2259 __wt_evict_file_exclusive_on() should clear WT_BTREE_NO_EVICTION on error
        WT-2260 Workloads evict internal pages unexpectedly
        WT-2262 Random sampling is skewed by tree shape
        WT-2265 Wiredtiger related change in ppc64le specific code block in gcc.h
        WT-2266 Add wtperf config to set if perf thresholds are fatal
        WT-2267 Improve wtperf throttling implementation to provide steady load
        WT-2269 wtperf should dump its config everytime it runs
        WT-2272 Stress test assertion in the sweep server
        WT-2275 broken DB after application crash
        WT-2276 tool to decode checkpoint addr
        WT-2277 Remove WT check against big-endian systems
        WT-2279 Define WT_PAUSE(), WT_FULL_BARRIER(), etc when s390x is defined
        WT-2281 wtperf smoke.sh fails on ppc64le
        WT-2282 error in wt_txn_update_oldest verbose message test
        WT-2283 retry in txn_update_oldest results in a hang
        WT-2284 Repeated macro definition
        WT-2285 configure should set BUFFER_ALIGNMENT_DEFAULT to 4kb on linux
        WT-2287 WT_SESSION.rebalance
        WT-2289 failure in fast key check
        WT-2290 WT_SESSION.compact could be more effective.
        WT-2291 Random cursor walk inefficient in skip list only trees
        WT-2295 WT_SESSION.create does a full-scan of the main table
        WT-2296 New log algorithm needs improving for sync/flush settings
        WT-2297 Fix off-by-one error in Huffman config file parsing
        WT-2299 upper-level WiredTiger code is reaching into the block manager
        WT-2301 Add reading a range to wtperf
        WT-2303 Build warning in wtperf
        WT-2304 wtperf crash dumping config
        WT-2305 Fix coverity scan issues on 23/12/2015
        WT-2307 Internal page splits can corrupt cursor iteration
        WT-2308 custom extractor for ref_cursors in join cursor
        WT-2311 Support Sparc
        WT-2312 re-creating a deleted column-store page can corrupt the in-memory tree
        WT-2313 sweep-server: conn_dhandle.c, 610: dhandle != conn->cache->evict_file_next
        WT-2314 page-swap error handling is inconsistent
        WT-2316 stress test failure: WT_CURSOR.prev out-of-order returns
        WT-2320 Only check copyright when cutting releases
        WT-2321 WT-2321: race between eviction and worker threads on the eviction queue
        WT-2326 Change WTPERF to use new memory allocation functions instead of the standard
        WT-2328 schema drop does direct unlink, it should use a block manager interface.
        WT-2331 Checking of search() result for reference cursors before join()
        WT-2332 Bug in logging write-no-sync mode
        WT-2333 Add a flag so drop doesn't block
        WT-2335 NULL pointer crash in config_check_search with invalid configuration string
        WT-2338 Disable using pre-allocated log files when backup cursor is open
        WT-2339 format post-rebalance verify failure (stress run #11586)
        WT-2340 Add logging guarantee assertions, whitespace
        WT-2342 Enhance wtperf to support background create and drop operations
        WT-2344 OS X compiler warning
        WT-2347 Java: schema format edge cases
        WT-2348 xargs -P isn't portable
        WT-2355 Fix minor scratch buffer usage in logging
        SERVER-21833 Compact does not release space to the system with WiredTiger
        SERVER-21887 $sample takes disproportionately long time on newly created collection
        SERVER-22064 Coverity analysis defect 77699: Unchecked return value
        SERVER-21944 WiredTiger changes for 3.2.2
        Branch: v3.2
        https://github.com/mongodb/mongo/commit/5d6532f3d5227ff76f62c4810c98a4ef4d0c8c56

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger-wiredtiger-2.7.0-559-g07966a4.tar.gz from wiredtiger branch mongodb-3.2 ref: 3c2ad56..07966a4 WT-1517 schema format edge cases WT-1801 Add a directory sync after rollback of a WT_SESSION::rename operation WT-2060 Simplify aggregation of statistics WT-2073 metadata cleanups WT-2099 Seeing memory underflow messages WT-2113 truncate01 sometimes fails WT-2142 Connection cleanup in Python tests WT-2177 Add an optional per-thread seed to random number generator WT-2198 bulk load and column store appends WT-2216 simplify row-store search loop slightly WT-2225 New split code performance impact WT-2231 pinned page cursor searches could check parent keys WT-2235 wt printlog option without unicode WT-2242 WiredTiger treats dead trees the same as other trees in eviction WT-2244 Trigger in-memory splits sooner WT-2245 WTPERF Truncate has no ability to catch up when it falls behind WT-2246 column-store append searches the leaf page; the maximum record number fails CRUD operations WT-2247 variable-length column-store in-memory page splits WT-2256 WTPERFs throttle option fires in bursts WT-2257 wtperf doesn't handle overriding workload config WT-2258 WiredTiger preloads pages even when direct-IO is configured. WT-2259 __wt_evict_file_exclusive_on() should clear WT_BTREE_NO_EVICTION on error WT-2260 Workloads evict internal pages unexpectedly WT-2262 Random sampling is skewed by tree shape WT-2265 Wiredtiger related change in ppc64le specific code block in gcc.h WT-2266 Add wtperf config to set if perf thresholds are fatal WT-2267 Improve wtperf throttling implementation to provide steady load WT-2269 wtperf should dump its config everytime it runs WT-2272 Stress test assertion in the sweep server WT-2275 broken DB after application crash WT-2276 tool to decode checkpoint addr WT-2277 Remove WT check against big-endian systems WT-2279 Define WT_PAUSE(), WT_FULL_BARRIER(), etc when s390x is defined WT-2281 wtperf smoke.sh fails on ppc64le WT-2282 error in wt_txn_update_oldest verbose message test WT-2283 retry in txn_update_oldest results in a hang WT-2284 Repeated macro definition WT-2285 configure should set BUFFER_ALIGNMENT_DEFAULT to 4kb on linux WT-2287 WT_SESSION.rebalance WT-2289 failure in fast key check WT-2290 WT_SESSION.compact could be more effective. WT-2291 Random cursor walk inefficient in skip list only trees WT-2295 WT_SESSION.create does a full-scan of the main table WT-2296 New log algorithm needs improving for sync/flush settings WT-2297 Fix off-by-one error in Huffman config file parsing WT-2299 upper-level WiredTiger code is reaching into the block manager WT-2301 Add reading a range to wtperf WT-2303 Build warning in wtperf WT-2304 wtperf crash dumping config WT-2305 Fix coverity scan issues on 23/12/2015 WT-2307 Internal page splits can corrupt cursor iteration WT-2308 custom extractor for ref_cursors in join cursor WT-2311 Support Sparc WT-2312 re-creating a deleted column-store page can corrupt the in-memory tree WT-2313 sweep-server: conn_dhandle.c, 610: dhandle != conn->cache->evict_file_next WT-2314 page-swap error handling is inconsistent WT-2316 stress test failure: WT_CURSOR.prev out-of-order returns WT-2320 Only check copyright when cutting releases WT-2321 WT-2321 : race between eviction and worker threads on the eviction queue WT-2326 Change WTPERF to use new memory allocation functions instead of the standard WT-2328 schema drop does direct unlink, it should use a block manager interface. WT-2331 Checking of search() result for reference cursors before join() WT-2332 Bug in logging write-no-sync mode WT-2333 Add a flag so drop doesn't block WT-2335 NULL pointer crash in config_check_search with invalid configuration string WT-2338 Disable using pre-allocated log files when backup cursor is open WT-2339 format post-rebalance verify failure (stress run #11586) WT-2340 Add logging guarantee assertions, whitespace WT-2342 Enhance wtperf to support background create and drop operations WT-2344 OS X compiler warning WT-2347 Java: schema format edge cases WT-2348 xargs -P isn't portable WT-2355 Fix minor scratch buffer usage in logging SERVER-21833 Compact does not release space to the system with WiredTiger SERVER-21887 $sample takes disproportionately long time on newly created collection SERVER-22064 Coverity analysis defect 77699: Unchecked return value SERVER-21944 WiredTiger changes for 3.2.2 Branch: v3.2 https://github.com/mongodb/mongo/commit/5d6532f3d5227ff76f62c4810c98a4ef4d0c8c56

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Days since reply:
              1 year, 7 weeks, 6 days ago
              Date of 1st Reply: