[SERVER-17329] Improve management of server version in build system Created: 19/Feb/15  Updated: 04/Nov/15  Resolved: 21/May/15

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 3.0.5, 3.1.4

Type: Improvement Priority: Major - P3
Reporter: Andrew Morrow (Inactive) Assignee: Jonathan Reams
Resolution: Done Votes: 2
Labels: build-planning
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-17484 Migrate server MCI config into server... Closed
Related
related to SERVER-18910 Path in distribution does not contain... Closed
is related to SERVER-19802 --version command-line flag should sh... Closed
is related to SERVER-19822 Zip file has wrong directory name Closed
is related to SERVER-21286 versionArray not computed correctly f... Closed
is related to SERVER-17782 Generate source tarballs with pre-int... Closed
Backwards Compatibility: Minor Change
Backport Completed:
Sprint: BUILD 0 3/13/15, BUILD 2 04/24/15, BUILD 4 06/05/15
Participants:

 Description   

The mechanism by which the logical version number and git hash propagates into build artifacts has several deficiencies and adverse consequences. By improving this system, we can simplify the build process, and lead to simpler to issue releases.

The following pieces of information are automatically determined inputs into the versioning subsystem:

  • The current githash
  • The currently selected JS engine
  • The currently selected allocator (system vs. tcmalloc)
  • The linker and compiler flags
  • System info (usually uname or the equivalent).

Additionally, the actual version string (like 3.1.0-pre) is hard coded into the following files:

  • src/mongo/util/version.cpp
  • doxygenConfig

At build time, all of these pieces of information are fed to custom SCons command in SConscript.buildinfo, which generates a buildinfo.cpp file into the current SCons variant dir, containing interpolated values.

This buildinfo.cpp file is then compiled, along with util/version.cpp into a low level library called $BUILD_DIR/mongo/version.a. The information available from the synthesis of these files is made available via the util/version.h file, as well as by the util/version_reporting.h header, which provides an overlapping set of versioning functions.

The implementation of the logical versioning functionality is itself split across the buildinfo.cpp file, the version.cpp file, and the version_reporting.cpp file.

This organization has the following consequences:

  • There is no build-time machine-readable representation of the version number. This means that SCons doesn't actually know the version it is building, so it can't act on/with it.
  • To produce a release, the release engineer must correctly edit two files, commit that change with the proper BUMP message (repeating the version). Once the release is bumped, the same edits and commit must be manually made for the 'post' commit.
  • The SConscript.buildinfo custom target uses its own templating mechanism to produce the buildinfo file, when it could use the builtin scons SubstFile mechanism.
  • The relationship between the various version headers, generated source files, and non-generated source files is very unclear.
  • Because the current git SHA is unconditionally interpolated into the buildinfo.cpp file, making or rebasing a commit causes a rebuild of buildinfo.cpp, forcing a re-link of every target. But adding the SHA to the build is not useful during daily development work.
  • Traditional _VERSION_MAJOR, _VERSION_MINOR macros are not defined in the version.h header. The information that is exposed is poorly organized and often not well formatted for real use cases.
  • Package spec files are also interpolated, but by a script run externally to the build system, requiring yet another manual step when producing a build, and again requiring that the version info be provided manually.
  • The continuous integration system must grep the version number out of the sources and parse it.

To address these issues, we should overhaul this entire system. Roughly:

  • There should be a single source of machine readable version truth. Presumably, this file should be at the root of the tree, be called version.json, and contain a JSON document containing, at minimum, fields describing the major, minor, path, and extra version values for the server.
  • There should be new build script to make the 'BUMP' and 'post' commits, which takes a user specified version string as the only argument, edits the root version.json file, after parsing and validating the structure of the version argument, and then commit these edits with an appropriately formatted commit message.
  • The root SConstruct file should be updated to load the current version.json file at startup and store the relevant variables (MONGO_VERSION_MAJOR, etc.) into the Environment.
  • The doxygenConfig and buildinfo.cpp files should be converted to be .in files, and should be populated via the SCons SubstFile mechanism with the relevant values from the environment.
  • Similar substitution should be used to inject relevant information into packaging .in files, and the packaging script should be re-factored into a SCons target that depends on the generated package files and invokes the RPM or deb build task appropriately.
  • The version.h, version_reporting.h, version_reporting.cpp, and buildinfo.cpp files should be re-organized to clarify the relationship between them. Ideally, there should be only one generated version.h header, and one non-generated implementation file that includes it. It may be necessary to generate both a header and a source file. To the extent possible, the generated files should be small.
  • A new SCons flag, something along the lines of --disable-build-githash should be provided that suppresses injecting the githash into the buildinfo.cpp file so that developers are not forced to re-link when making local commits.

Overall, making changes along the lines outlined above would simplify the BUMP and post process, clean up the relationship between the various versioning artifacts, speed up developer builds, improve the automation of packaging, and reduce our use of custom code generation that duplicates existing SCons features.



 Comments   
Comment by Githook User [ 15/Jul/15 ]

Author:

{u'username': u'ehershey', u'name': u'Ernie Hershey', u'email': u'ernie.hershey@10gen.com'}

Message: Use version.json to verify 3.0 versions now that SERVER-17329 is in 3.0
Branch: master
https://github.com/10gen/kernel-tools/commit/d66af60ac5076ac408ef3c43db5fede1236d0bf8

Comment by Daniel Pasette (Inactive) [ 14/Jul/15 ]

This was committed to v3.0 here:

commit 180a24d624984c7994ff52862699d45ebc609dc1
Author: Jonathan Reams <jbreams@mongodb.com>
Date:   Thu May 21 14:15:43 2015 -0400
 
    BUILD-770 Backport version info and source tarball changes

and here:

commit dd08d48c0e4aaa6679419132aa85f482c16921d2
Author: Jonathan Reams <jbreams@mongodb.com>
Date:   Sun Jul 12 14:15:06 2015 -0400
 
    BUILD-770 Fix windows-64/windows-32 compiles

Comment by Ernie Hershey [ 16/Jun/15 ]

Jonathan and I just talked about backporting this - 3.0 for sure and Jonathan will figure out how hard/sensible backporting to 2.6 would be.

Comment by Githook User [ 21/May/15 ]

Author:

{u'username': u'jbreams', u'name': u'Jonathan Reams', u'email': u'jbreams@mongodb.com'}

Message: SERVER-17782 SERVER-17329 Improve versioning and add distsrc to SCons
Branch: master
https://github.com/mongodb/mongo/commit/2ce0a91ec6c28f6c4b401b852a157977b709840e

Comment by Githook User [ 10/Mar/15 ]

Author:

{u'username': u'acmorrow', u'name': u'Andrew Morrow', u'email': u'acm@mongodb.com'}

Message: SERVER-17329 Fix version.txt management for rocks build
Branch: master
https://github.com/mongodb/mongo/commit/f974b0e9639199b3f41dfcb38358859290343c55

Comment by Githook User [ 09/Mar/15 ]

Author:

{u'username': u'acmorrow', u'name': u'Andrew Morrow', u'email': u'acm@mongodb.com'}

Message: SERVER-17329 Interpolate new version.txt file into generated buildinfo files
Branch: master
https://github.com/mongodb/mongo/commit/055d827ed7c144f26441441a75ea8dd5f797f9de

Generated at Thu Feb 08 03:44:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.