Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34374

resmoke.py uses bytestrings for representing pathnames, leading to silently failing to clear the dbpath on Windows

    • Fully Compatible
    • ALL
    • v3.6, v3.4, v3.2
    • TIG 2018-05-07, TIG 2018-05-21
    • 16
    • 2

      https://bugs.python.org/issue24672 describes an issue in Python where shutil.rmtree() fails to delete files with non-ASCII pathnames when a bytestring (i.e. a str instance in Python 2). The ntpath.py module in Python preserves type of its argument so it sufficient to use a unicode instance instead in order to have Python call the W-suffixed Win32 APIs that return Unicode strings.

      I've verified on a Windows spawn host that the following patch to config.py addresses this issue. The change to parser.py is to just do the same if someone were to specify --dbpathPrefix when trying to reproduce a failure outside of Evergreen.

      Unable to find source-code formatter for language: diff. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      diff --git a/buildscripts/resmokelib/config.py b/buildscripts/resmokelib/config.py
      index 66753c389d..2f13c2df96 100644
      --- a/buildscripts/resmokelib/config.py
      +++ b/buildscripts/resmokelib/config.py
      @@ -34,7 +34,7 @@ DEFAULT_BENCHMARK_MIN_TIME = datetime.timedelta(seconds=5)
       # Default root directory for where resmoke.py puts directories containing data files of mongod's it
       # starts, as well as those started by individual tests.
      -DEFAULT_DBPATH_PREFIX = os.path.normpath("/data/db")
      +DEFAULT_DBPATH_PREFIX = os.path.normpath(u"/data/db")
       # Names below correspond to how they are specified via the command line or in the options YAML file.
       DEFAULTS = {
      diff --git a/buildscripts/resmokelib/parser.py b/buildscripts/resmokelib/parser.py
      index d9f40da3e9..1353f899fd 100644
      --- a/buildscripts/resmokelib/parser.py
      +++ b/buildscripts/resmokelib/parser.py
      @@ -352,7 +352,7 @@ def update_config_vars(values):  # pylint: disable=too-many-statements
           _config.ARCHIVE_LIMIT_TESTS = config.pop("archive_limit_tests")
           _config.BASE_PORT = int(config.pop("base_port"))
           _config.BUILDLOGGER_URL = config.pop("buildlogger_url")
      -    _config.DBPATH_PREFIX = _expand_user(config.pop("dbpath_prefix"))
      +    _config.DBPATH_PREFIX = unicode(_expand_user(config.pop("dbpath_prefix")))
           _config.DBTEST_EXECUTABLE = _expand_user(config.pop("dbtest_executable"))
           _config.DRY_RUN = config.pop("dry_run")
           _config.EXCLUDE_WITH_ANY_TAGS = _tags_from_list(config.pop("exclude_with_any_tags"))

      However, I'm not sure if more special handling on Linux platforms is necessary as the changes from https://github.com/pypa/setuptools/commit/5ad13718686bee04a93b4e86929c1bb170f14a52 suggest we shouldn't use Unicode string literals if sys.getfilesystemencoding() == 'ascii'. We currently set the LANG=C environment variable on all of Ubuntu 16.04 builders (SERVER-31717, SERVER-33184) so it isn't clear why we'd even be able to create files with non-ASCII pathnames. CC mark.benvenuto

      $ LANG=C python -c 'import sys; print(sys.getfilesystemencoding())'

            jonathan.abrahams Jonathan Abrahams
            max.hirschhorn@mongodb.com Max Hirschhorn
            0 Vote for this issue
            3 Start watching this issue