[CDRIVER-2416] Merge libbson repository into mongo-c-driver Created: 05/Dec/17  Updated: 28/Oct/23  Resolved: 21/May/18

Status: Closed
Project: C Driver
Component/s: Build
Affects Version/s: None
Fix Version/s: 1.10.0

Type: Epic Priority: Major - P3
Reporter: A. Jesse Jiryu Davis Assignee: Roberto Sanchez
Resolution: Fixed Votes: 0
Labels: FY2019Q2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by CDRIVER-2509 Check that spec tests are up to date Closed
Problem/Incident
causes CDRIVER-2695 Share threading config checks between... Closed

 Description   

Currently libbson is in a separate repository from libmongoc; libmongoc includes it as a Git submodule. This causes various problems. It's difficult to build libmongoc with a bundled checkout of libbson using CMake (CDRIVER-1948). It's confusing to develop new libbson features that libmongoc depends on. Common logic in the test suite and build systems must be duplicated across the two repositories, and inevitably the common logic diverges.

Replace the git submodule in libmongoc with the contents of the libbson repository and retire the libbson repository. Do this as a fake author to avoid biasing the GitHub contributor stats.

Merge the two test suites into one: no more test-libbson executable, test-libmongoc should now run all the tests for both libraries.

Factor the common build logic. If we haven't yet deleted the Autotools scripts (CDRIVER-1349) don't bother factoring the duplicated Autotools logic, only do so for CMake scripts.

Continue to ship libbson and libmongoc as distinct libraries; only the repositories are merged.



 Comments   
Comment by Roberto Sanchez [ 21/May/18 ]

Woohoo!

Comment by A. Jesse Jiryu Davis [ 21/May/18 ]

Yes, go ahead and close it!

Comment by Roberto Sanchez [ 21/May/18 ]

jesse, kevin.eric.albertson@gmail.com, are we ready/able to close this Epic now that all of of the constituent tasks are complete?

Comment by Githook User [ 15/Apr/18 ]

Author:

{'name': 'Roberto C. Sánchez', 'email': 'roberto@connexer.com', 'username': 'rcsanchez97'}

Message: CDRIVER-2416 implement ENABLE_BSON=ONLY build option
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/e025efb85d00894350349b4dd83461c2b17349bc

Comment by Githook User [ 20/Feb/18 ]

Author:

{'email': 'roberto@connexer.com', 'name': 'Roberto C. Sánchez', 'username': 'rcsanchez97'}

Message: CDRIVER-2416 consolidate redundant cmake logic
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/831f13c90e52d7d5fb02f3967cabda1cfdbcac8f

Comment by Githook User [ 30/Jan/18 ]

Author:

{'email': 'jesse@mongodb.com', 'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis'}

Message: CDRIVER-2416 fix libbson tests on darwin
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/e7a9a5f79691810a30cdac57475c0c7914269520

Comment by Githook User [ 29/Jan/18 ]

Author:

{'email': 'jesse@mongodb.com', 'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis'}

Message: CDRIVER-2416 generate libbson docs in Evergreen
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/9213b827ff3c1f5648e9e62451f5ea06847bac4e

Comment by Githook User [ 29/Jan/18 ]

Author:

{'email': 'jesse@mongodb.com', 'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis'}

Message: CDRIVER-2416 obsolete libbson config files
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/450a6d60006dd4477aecb93dae6f1b4bf3e04696

Comment by Githook User [ 29/Jan/18 ]

Author:

{'email': 'jesse@mongodb.com', 'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis'}

Message: CDRIVER-2416 test bundled libbson
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/caf1839cd9238f8c881d720b9335c48c634b1d5e

Comment by Githook User [ 28/Jan/18 ]

Author:

{'email': 'jesse@mongodb.com', 'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis'}

Message: CDRIVER-2416 use bundled libbson in tests
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/835db65aec0501e2048dc63a49825c18737ed515

Comment by Githook User [ 28/Jan/18 ]

Author:

{'email': 'jesse@mongodb.com', 'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis'}

Message: CDRIVER-2416 build bundled libbson with cmake
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/2d4c9f5c3e950aa10e8ae4d37f6a82bb01c5ac66

Comment by Githook User [ 28/Jan/18 ]

Author:

{'email': 'jesse@mongodb.com', 'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis'}

Message: CDRIVER-2416 move libbson dist-checks to libmongoc
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/67c88d2a49ed4f185a05ab081d04d710fadcd90b

Comment by Githook User [ 28/Jan/18 ]

Author:

{'email': 'jesse@mongodb.com', 'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis'}

Message: CDRIVER-2416 fix "make dist" for merged repos
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/144e60f77501aed1889df7d4208c0bdda36d8e64

Comment by Roberto Sanchez [ 16/Jan/18 ]

I have updated the PR with a change that produces a properly versioned libmongoc -> libbson dependency.

Comment by Roberto Sanchez [ 12/Jan/18 ]

OK. I will make the change and update the PR.

Comment by A. Jesse Jiryu Davis [ 12/Jan/18 ]

It sounds like we should make the dependency versioned, yes. Thanks for catching this.

Comment by Roberto Sanchez [ 12/Jan/18 ]

jesse, I wanted to bring to your attention regarding the libbson/libmongoc dependencies. In Debian, libmongoc-1.0-0 depends on libbson-1.0-0, but the dependency is unversioned:

https://packages.debian.org/sid/libmongoc-1.0-0

The dependencies of libmongoc-dev -> libbson-dev,libmongoc-1.0-0 and libbson-dev -> libbson-1.0-0 are properly versioned because we explicitly state them in the control file. However, the the dependency libmongoc-1.0-0 -> libbson-1.0-0 is calculated by the dh_shlibdeps helper script. Since we don't do symbol versioning and don't ship a shlibs file, the dependency is unversioned. This isn't necessarily a bad thing, but there is a theoretical possibility that an upgrade could be performed to libmongoc-1.0-0 without a corresponding upgrade to libbson-1.0-0.

I think that with the merged repositories it is even less likely that a problem will occur with this because both binary packages will now originate from the same source package. However, it is a bit easier to fix this now because we have a guarantee that both packages will have the same version number because we build from the same source package. We can override the call to dh_makeshlibs during the package build and supply version information to make the dependency versioned.

To summarize, if we don't make a change regarding this then the dependencies will continue largely as they have been up to now. The reason I bring this up now is that if we want to change it the approach is more straightforward than it would have been under the split repository setup.

Comment by A. Jesse Jiryu Davis [ 09/Jan/18 ]

Sounds great.

Comment by Roberto Sanchez [ 09/Jan/18 ]

jesse, I have made what I believe are all the necessary changes. With the rewind recipe you gave me I will be able to simulate a 1.10.0-dev release to build a test package and confirm that everything is in order. My plan is that once I have it tested and I am happy with the Debian packaging I will submit a pull request with the changes. That will allow you to sit on the pull request until you are ready for a 1.10.0 final release. That way, if we need to make a 1.9.1 release we aren't fighting the packaging changes for 1.10.0.

Comment by A. Jesse Jiryu Davis [ 09/Jan/18 ]

Hi! I've read your explanation and it seems wise. Please proceed according to your plan.

Comment by Roberto Sanchez [ 09/Jan/18 ]

jesse, I have put some thought into this today and I would like to propose a simpler solution than the multiple source tarballs.

The bottom line is that if you are OK with the upstream and Debian release tarballs being different today and you are OK with them continuing to be different in the future, then there is no reason we can't just make some minor tweaks to libmongoc Debian packaging and be done with it. Read on for all the gory details.

It occurred to me that the current process for generating upstream libbson/libmongoc releases and the process for generating their Debian packages results in two different source tarballs for the same release. The reason for this is that you use some process (maybe 'make dist' or similar) to produce a release tarball and the Debian package process relies the contents of the 'upstream' branch (which is called 'r1.9' in the repository). For example, on master, there is no Makefile or configure script, as the build has to be bootstrapped. The release tarball that you ship includes those items. As a result, if someone downloads your release tarball from GitHub and the .orig.tar.gz for the same upstream release from the Debian package, they would find that the two files had different MD5 sums and different contents. This is not necessarily bad, it just is.

In the typical packaging workflow, upstream and Debian packaging develop in separate repositories. The normal workflow has the Debian maintainer import the upstream release tarball into a Debian packaging-specific repository. This is usually done with the assistance of pristine-tar to ensure that the Debian package building process outputs a bit-for-bit identical original tarball as what originated from upstream.

However, in our case we share a repository and we just let the Debian package scripts generate a tarball from the point on the upstream branch which is identified in gbp.conf. Since we do not import a tarball with pristine-tar enabled, there is no reference in terms of a specific file to try and match. Personally, I think that the current workflow is good from that standpoint and I think it would be far more trouble than it is worth to try to create a Debian .orig.tar.gz that matches the release tarball you create for upstream release from GitHub.

The point of all that preamble is to get to this: since we already produce different release tarballs for upstream releases and Debian packages, we can just create a single libmongoc Debian tarball that contains the sources for both libmongoc and libbson. The whole purpose of multiple upstream tarballs is to allow a Debian source package to be made up of multiple components which are released independently by upstream. However, since the Debian packaging here doesn't use those independent releases and we work on Debian packaging from within a repository that has all the sources together in one place, I think that the right answer is to just migrate the necessary files from the libbson Debian packaging directory into the libmongoc Debian packaging directory. Specifically, the .install, .docs, .lintian-overrides, shlibs, and possibly one or two other. The paths would likely need to be adjusted and I would make sure that the correct build switches and flags get carried through to the libbson build.

That would be a whole lot simpler than trying to mess with multiple release tarballs in the packaging process. Trying to do that would require a considerable change to the packaging workflow that we have established and I think that it would actually be a change for the worse.

Comment by A. Jesse Jiryu Davis [ 08/Jan/18 ]

Hi Roberto. Yes please do that Debian work next, and please see if there's any shared logic in the two CMakeLists.txt that's worth factoring into shared cmake include files. (Maybe there isn't.)

After that, we should factor together the test framework code, but I might be a better person for that work, compared to your skills which are more valuable elsewhere.

Comment by Roberto Sanchez [ 08/Jan/18 ]

jesse, I would like to make sure that I have a clear understanding of what I need to accomplish in order to resolve this ticket. Is the intent that I now produce a merged Debian package from the merged repository, or was there something else in addition to or instead of that?

Comment by Githook User [ 04/Jan/18 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'username': 'ajdavis', 'email': 'jesse@mongodb.com'}

Message: CDRIVER-2416 test libbson in evergreen

Fix libbson's Evergreen test scripts now that its repository is merged
into mongo-c-driver.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/10767de6fad9ec2947746bd342ba450633d9d2a5

Comment by Githook User [ 14/Dec/17 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@emptysquare.net', 'username': 'ajdavis'}

Message: CDRIVER-2416 update README
Branch: master
https://github.com/mongodb/libbson/commit/09b809083fcaca11be0c504d90d06e6ccdd4cdaf

Comment by Githook User [ 14/Dec/17 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2416 fix symlink to libbson's README

libbson's autogen.sh tries to ln -s README.md README, but libbson has
no README.md, it has README.rst. It's surprising that libbson has been
able to generate its release tarball until now with this bug, but for
some reason it's only a problem now that libbson is merged in.
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/d3490a1ed90e47bf095fe3825e33d944291ebc23

Comment by Githook User [ 14/Dec/17 ]

Author:

{'name': 'Merge libbson', 'email': 'drivers@mongodb.com'}

Message: CDRIVER-2416 add libbson as subdirectory
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/91dca46cf53be09afb305d65db420793b72120dc

Comment by Githook User [ 14/Dec/17 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2416 don't .gitignore src/libbson/COPYING
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/68185daa6492ff99c7498e17ba9d42b887772918

Comment by Githook User [ 14/Dec/17 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: CDRIVER-2416 delete libbson submodule
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/a13cb8699c5dbdfff4014c57c8a9369319475ecf

Comment by A. Jesse Jiryu Davis [ 05/Dec/17 ]

From Roberto:

Here is some info having multiple upstream tarballs in a Debian package:

https://wiki.debian.org/Projects/DebSrc3.0#How_to_use_multiple_upstream_tarballs_in_3.0_.28quilt.29_format.3F
https://raphaelhertzog.com/2010/09/07/how-to-use-multiple-upstream-tarballs-in-debian-source-packages/

I think that the tarballs would be named libmongoc_1.8.1.orig.tar.gz and
libmongoc_1.8.1.orig-libbson.tar.gz.

Generated at Wed Feb 07 21:15:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.