Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17329

Improve management of server version in build system

    • Type: Icon: Improvement Improvement
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.0.5, 3.1.4
    • Affects Version/s: None
    • Component/s: Build
    • Labels:
    • Minor Change
    • BUILD 0 3/13/15, BUILD 2 04/24/15, BUILD 4 06/05/15

      The mechanism by which the logical version number and git hash propagates into build artifacts has several deficiencies and adverse consequences. By improving this system, we can simplify the build process, and lead to simpler to issue releases.

      The following pieces of information are automatically determined inputs into the versioning subsystem:

      • The current githash
      • The currently selected JS engine
      • The currently selected allocator (system vs. tcmalloc)
      • The linker and compiler flags
      • System info (usually uname or the equivalent).

      Additionally, the actual version string (like 3.1.0-pre) is hard coded into the following files:

      • src/mongo/util/version.cpp
      • doxygenConfig

      At build time, all of these pieces of information are fed to custom SCons command in SConscript.buildinfo, which generates a buildinfo.cpp file into the current SCons variant dir, containing interpolated values.

      This buildinfo.cpp file is then compiled, along with util/version.cpp into a low level library called $BUILD_DIR/mongo/version.a. The information available from the synthesis of these files is made available via the util/version.h file, as well as by the util/version_reporting.h header, which provides an overlapping set of versioning functions.

      The implementation of the logical versioning functionality is itself split across the buildinfo.cpp file, the version.cpp file, and the version_reporting.cpp file.

      This organization has the following consequences:

      • There is no build-time machine-readable representation of the version number. This means that SCons doesn't actually know the version it is building, so it can't act on/with it.
      • To produce a release, the release engineer must correctly edit two files, commit that change with the proper BUMP message (repeating the version). Once the release is bumped, the same edits and commit must be manually made for the 'post' commit.
      • The SConscript.buildinfo custom target uses its own templating mechanism to produce the buildinfo file, when it could use the builtin scons SubstFile mechanism.
      • The relationship between the various version headers, generated source files, and non-generated source files is very unclear.
      • Because the current git SHA is unconditionally interpolated into the buildinfo.cpp file, making or rebasing a commit causes a rebuild of buildinfo.cpp, forcing a re-link of every target. But adding the SHA to the build is not useful during daily development work.
      • Traditional _VERSION_MAJOR, _VERSION_MINOR macros are not defined in the version.h header. The information that is exposed is poorly organized and often not well formatted for real use cases.
      • Package spec files are also interpolated, but by a script run externally to the build system, requiring yet another manual step when producing a build, and again requiring that the version info be provided manually.
      • The continuous integration system must grep the version number out of the sources and parse it.

      To address these issues, we should overhaul this entire system. Roughly:

      • There should be a single source of machine readable version truth. Presumably, this file should be at the root of the tree, be called version.json, and contain a JSON document containing, at minimum, fields describing the major, minor, path, and extra version values for the server.
      • There should be new build script to make the 'BUMP' and 'post' commits, which takes a user specified version string as the only argument, edits the root version.json file, after parsing and validating the structure of the version argument, and then commit these edits with an appropriately formatted commit message.
      • The root SConstruct file should be updated to load the current version.json file at startup and store the relevant variables (MONGO_VERSION_MAJOR, etc.) into the Environment.
      • The doxygenConfig and buildinfo.cpp files should be converted to be .in files, and should be populated via the SCons SubstFile mechanism with the relevant values from the environment.
      • Similar substitution should be used to inject relevant information into packaging .in files, and the packaging script should be re-factored into a SCons target that depends on the generated package files and invokes the RPM or deb build task appropriately.
      • The version.h, version_reporting.h, version_reporting.cpp, and buildinfo.cpp files should be re-organized to clarify the relationship between them. Ideally, there should be only one generated version.h header, and one non-generated implementation file that includes it. It may be necessary to generate both a header and a source file. To the extent possible, the generated files should be small.
      • A new SCons flag, something along the lines of --disable-build-githash should be provided that suppresses injecting the githash into the buildinfo.cpp file so that developers are not forced to re-link when making local commits.

      Overall, making changes along the lines outlined above would simplify the BUMP and post process, clean up the relationship between the various versioning artifacts, speed up developer builds, improve the automation of packaging, and reduce our use of custom code generation that duplicates existing SCons features.

            jonathan.reams@mongodb.com Jonathan Reams
            andrew.morrow@mongodb.com Andrew Morrow (Inactive)
            2 Vote for this issue
            8 Start watching this issue