Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7603

Sort statistics to fix JSON output

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT10.0.1, 4.4.8, 5.0.2, 5.1.0-rc0
    • Component/s: None
    • Labels:
      None
    • Story Points:
      5
    • Sprint:
      Storage - Ra 2021-07-12
    • Backport Requested:
      v4.4

      Description

      I noticed this while trying to get t2 data from test format, a lot of the stats I was interested in were missing.

      It turns out that when we output our statistics JSON we don't sort the sets. So we have duplicate cache, cursor, reconciliation and transaction keys (possibly more) in the JSON. The net effect being that when converted into a proper json by a tool, e.g. t2 the first duplicated key is usually deleted. As such we lose some statistics.

      There's a few possible fixes here but the one I implemented to test is simply sorting the connection statistics (the same will need to be done for the datasource stats):

       diff --git a/dist/stat.py b/dist/stat.py
      index 2e0cb0a86..99624fa91 100644
      --- a/dist/stat.py
      +++ b/dist/stat.py
      @@ -3,13 +3,14 @@ import re, string, sys, textwrap
       from dist import compare_srcfile, format_srcfile
      -
      +from functools import cmp_to_key
       # Read the source files.
       from stat_data import groups, dsrc_stats, connection_stats, conn_dsrc_stats, join_stats, \
      -    session_stats
      +    session_stats, Stat connection_statistics = connection_stats
       connection_statistics.extend(conn_dsrc_stats)
      +connection_statistics = sorted(connection_statistics, key=cmp_to_key(Stat.make_comparator(Stat.__cmp__)))
       dsrc_statistics = dsrc_stats
       dsrc_statistics.extend(conn_dsrc_stats)diff --git a/dist/stat_data.py b/dist/stat_data.py
      index 89a5578b3..2cb718ebb 100644
      --- a/dist/stat_data.py
      +++ b/dist/stat_data.py
      @@ -29,7 +29,17 @@ class Stat:
               self.flags = flags     def __cmp__(self, other):
      -        return cmp(self.desc.lower(), other.desc.lower())
      +        return self.desc.lower() < other.desc.lower()
      +
      +    def make_comparator(less_than):
      +        def compare(x, y):
      +            if less_than(x, y):
      +                return -1
      +            elif less_than(y, x):
      +                return 1
      +            else:
      +                return 0
      +        return compare
      

       

        Attachments

        1. t2 broken.png
          t2 broken.png
          47 kB
        2. t2 fixed.png
          t2 fixed.png
          150 kB
        3. WiredTigerStat.22.12.broken
          22 kB
        4. WiredTigerStat.22.13.fixed
          22 kB

          Activity

            People

            Assignee:
            etienne.petrel Etienne Petrel
            Reporter:
            luke.pearson Luke Pearson
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: