Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7853

Validate the update chain before we attempt to insert it into the history store

    • 8
    • Storage - Ra 2021-09-06, Storage - Ra 2021-09-20
    • v5.0, v4.4

      unit-test-zstd failed on RHEL 8.0

      Host: ec2-3-231-24-193.compute-1.amazonaws.com
      Project: WiredTiger (develop)
      Commit: diff: WT-7832 Add an encryptor extension that uses the libsodium cryptography library. (#6780)

      • Add an encryptor extension that uses the libsodium cryptography library.

      It should really be audited by a cryptographer before being used, but
      is expected to be usable with at worst minor adjustments.

      It uses the aead_xchacha20poly1305_ietf construction from libsodium to
      encrypt and checksum blocks.

      It does not support retrieving keys from a key manager, there not
      being any obvious open-source choices that I'm aware of. This means
      that it can, for the time being anyway, only be configured with
      secretkey= and not keyid=, which is perhaps unfortunate but better
      than nothing.

      Besides the encryptor itself, this changeset includes the following
      related changes:

      • Add the new extension to both the cmake and autotools builds.
      • Rework the encryption page in the documentation, adding the new
        encryptor and expanding on some of the other material, and also
        add some bits/make some improvements to the WT_ENCRYPTOR docs.
      • In util_main.c, add a wt_explicit_zero function for zeroing
        memory that takes precautions against being removed by the
        compiler, and use it to clear copies of the secret key. Zero and
        free the secret key (and open config string, which contains the
        secret key when there is one) earlier.
      • In nop_encryptor.c, since this is supposed to be a template for
        application developers to fill in, add a blank customize method.
        Without a customize method you can't configure keys, so even
        though it's officially optional it seems like the example should
        have one.
      • Add support for the new extension to:
        • test_format
        • test_encrypt0[126].py
        • test_import0[69].py
          (Note that test_encrypt05 doesn't exist, test_encrypt0[34] are
          for testing the config plumbing and not any particular extension,
          and test_encrypt07 needs to be able to munge the "encrypted" data
          and doesn't work with real encryption.)
      • Add new test_encrypt0[89].py that checks the error paths in the
        new extension's customize method.
      • Add an example snippet for how to configure the new extension to
        ex_all.c for use in the docs.
      • Add the encryptor directory to Doxyfile so it can be an example.
      • Add the new encryptor to the examples page in the documentation.
      • Add a bunch of spelling words.
      • Add some of the functions to the exception list in s_void (like
        other extensions).

      It also includes the following change that is not related but directly
      adjacent to a piece of the above:

      • In the cmake build of test_format, pass the path to the zstd
        library with -D, like the other extensions.
      • Some minor adjustments from a preliminary review.
      • Document that WT's checksums can be disabled when using encryption.

      Because any viable encryptor applies a cryptographically strong
      checksum, there's no need to add a separate weaker checksum as well.
      Document this in the encryptors page and in the checksum argument of

      • Fix compiler warnings, missed by accident.
      • Initial changes from review.

      Also I missed something: the change in wiredtiger.in about configuring
      checksums also needs to be in api_data.py, and incurs another spelling

      • Argue with clang-format to get rid of the hanging-indent comments.
      • Make a couple more comment adjustments.
      • Try again with the comment formatting.

      It seems that the header is required to use hanging indent by
      function.py, so in order to avoid the rest of the comments after being
      reformatted with hanging indent by clang-format, move them inside the
      function body.

      This is maybe not optimal but it at least isn't visually revolting and
      doesn't break the tree :-|

      Also, add sodium_encrypt.c to dist/extlist so that all the checks are
      run on it.

      • Split the cleanup path for secretkey/p in two.

      Hopefully avoids false positives from inadequately path-sensitive
      static analyzers. | 15 Jul 21 01:30 UTC
      Evergreen Subscription: ; Evergreen Event:

      Task Logs (unit-test-zstd)

        1. OOO-MM.jpg
          118 kB
        2. Reconciliation.jpg
          84 kB

            luke.pearson@mongodb.com Luke Pearson
            xgen-evg-user Xgen-Evergreen-User
            Sid Mahajan
            0 Vote for this issue
            22 Start watching this issue