[SERVER-49457] Ninja mongod build failing with duplicate symbol linker error Created: 10/Jul/20  Updated: 29/Oct/23  Resolved: 14/Aug/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.4.1, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Hirday Gupta (Inactive) Assignee: Daniel Moody
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File build.ninja    
Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Steps To Reproduce:

Run ninja -j400 install-core at commit 6c0ac0678f3c159c65c497acf155e964dec43f61 on master branch of mongodb/mongo.

Sprint: Dev Platform 2020-07-27, Dev Platform 2020-08-10, Dev Platform 2020-08-24
Participants:

 Description   

Got a duplicate symbol error for some functions in WaitForMajorityServiceWaitForMajorityService. 

This is the SCONS command I had used (note that I did not regenerate Scons after git pulling):

./buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_v3_clang.vars --ninja generate-ninja ICECC=icecc CCACHE=ccache

And the Ninja command I invoked: 

ninja -j400 install-core

I was at git commit 6c0ac0678f3c159c65c497acf155e964dec43f61 on branch master of mongodb/mongo.

I've also attached my build.ninja file to the ticket.

Here's the full error:

 FAILED: build/opt/mongo/mongod 
 PATH=/opt/mongodbtoolchain/v3/bin:/usr/local/bin:/opt/bin:/bin:/usr/bin ICECC_CLANG_REMOTE_CPP=1 CCACHE_NOCPP2=1 ICECC_VERSION=/home/ubuntu/mongo/build/scons/icecc/opt_mongodbtoolchain_v3_bin_clang_opt_mongodbtoolchain_v3_bin_clang++.tar.gz CCACHE_PREFIX=/usr/bin/icecc /usr/bin/icerun /opt/mongodbtoolchain/v3/bin/clang++ @build/opt/mongo/mongod.rsp
 ld.lld: error: duplicate symbol: mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&)
 >>> defined at wait_for_majority_service.cpp:101 (src/mongo/db/s/wait_for_majority_service.cpp:101)
 >>> wait_for_majority_service.o:(mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&)) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
 >>> defined at wait_for_majority_service.cpp:101 (src/mongo/db/repl/wait_for_majority_service.cpp:101)
 >>> wait_for_majority_service.o:(.text+0x510) in archive build/opt/mongo/db/repl/libwait_for_majority_service.ald.lld: error: duplicate symbol: mongo::WaitForMajorityService::_periodicallyWaitForMajority(mongo::ServiceContext*)
 >>> defined at wait_for_majority_service.cpp:145 (src/mongo/db/s/wait_for_majority_service.cpp:145)
 >>> wait_for_majority_service.o:(mongo::WaitForMajorityService::_periodicallyWaitForMajority(mongo::ServiceContext*)) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
 >>> defined at wait_for_majority_service.cpp:145 (src/mongo/db/repl/wait_for_majority_service.cpp:145)
 >>> wait_for_majority_service.o:(.text+0xA50) in archive build/opt/mongo/db/repl/libwait_for_majority_service.ald.lld: error: duplicate symbol: mongo::WaitForMajorityService::get(mongo::ServiceContext*)
 >>> defined at wait_for_majority_service.cpp:61 (src/mongo/db/s/wait_for_majority_service.cpp:61)
 >>> wait_for_majority_service.o:(mongo::WaitForMajorityService::get(mongo::ServiceContext*)) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
 >>> defined at wait_for_majority_service.cpp:61 (src/mongo/db/repl/wait_for_majority_service.cpp:61)
 >>> wait_for_majority_service.o:(.text+0x3C0) in archive build/opt/mongo/db/repl/libwait_for_majority_service.ald.lld: error: duplicate symbol: mongo::WaitForMajorityService::setUp(mongo::ServiceContext*)
 >>> defined at wait_for_majority_service.cpp:65 (src/mongo/db/s/wait_for_majority_service.cpp:65)
 >>> wait_for_majority_service.o:(mongo::WaitForMajorityService::setUp(mongo::ServiceContext*)) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
 >>> defined at wait_for_majority_service.cpp:65 (src/mongo/db/repl/wait_for_majority_service.cpp:65)
 >>> wait_for_majority_service.o:(.text+0x3D0) in archive build/opt/mongo/db/repl/libwait_for_majority_service.ald.lld: error: duplicate symbol: mongo::WaitForMajorityService::shutDown()
 >>> defined at wait_for_majority_service.cpp:73 (src/mongo/db/s/wait_for_majority_service.cpp:73)
 >>> wait_for_majority_service.o:(mongo::WaitForMajorityService::shutDown()) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
 >>> defined at wait_for_majority_service.cpp:73 (src/mongo/db/repl/wait_for_majority_service.cpp:73)
 >>> wait_for_majority_service.o:(.text+0x130) in archive build/opt/mongo/db/repl/libwait_for_majority_service.ald.lld: error: duplicate symbol: mongo::WaitForMajorityService::~WaitForMajorityService()
 >>> defined at wait_for_majority_service.cpp:57 (src/mongo/db/s/wait_for_majority_service.cpp:57)
 >>> wait_for_majority_service.o:(mongo::WaitForMajorityService::~WaitForMajorityService()) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
 >>> defined at wait_for_majority_service.cpp:57 (src/mongo/db/repl/wait_for_majority_service.cpp:57)
 >>> wait_for_majority_service.o:(.text+0x0) in archive build/opt/mongo/db/repl/libwait_for_majority_service.ald.lld: error: duplicate symbol: mongo::WaitForMajorityService::~WaitForMajorityService()
 >>> defined at wait_for_majority_service.cpp:57 (src/mongo/db/s/wait_for_majority_service.cpp:57)
 >>> wait_for_majority_service.o:(mongo::WaitForMajorityService::~WaitForMajorityService()) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
 >>> defined at wait_for_majority_service.cpp:57 (src/mongo/db/repl/wait_for_majority_service.cpp:57)
 >>> wait_for_majority_service.o:(.text+0x0) in archive build/opt/mongo/db/repl/libwait_for_majority_service.a
 clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
 ninja: build stopped: subcommand failed.



 Comments   
Comment by Githook User [ 13/Oct/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50363 Merge --build-tools=next into stable

Merging the following fixes into the stable version of the build tools
and migrate fully from --ninja=next to the --build-tools=next:

(cherry picked from commit 78bb3f3c8a658a5a9fec8d55864e426382f68bd0)
Branch: v4.4
https://github.com/mongodb/mongo/commit/e7f6696a96db5438a269c5e67a06ff9e05cb897a

Comment by Githook User [ 07/Oct/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50363 Merge --build-tools=next into stable

Merging the following fixes into the stable version of the build tools
and migrate fully from --ninja=next to the --build-tools=next:

(cherry picked from commit 78bb3f3c8a658a5a9fec8d55864e426382f68bd0)
Branch: v4.4
https://github.com/mongodb/mongo/commit/7b93770e0835087c023ae3e3f36eeb46d204767f

Comment by Githook User [ 06/Oct/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50363 Merge --build-tools=next into stable

Merging the following fixes into the stable version of the build tools
and migrate fully from --ninja=next to the --build-tools=next:

Comment by Githook User [ 15/Aug/20 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-49457 Ninja tool static links should delete existing archive before linking"
Branch: v4.4
https://github.com/mongodb/mongo/commit/3bfeaa9d487709a2169aa57d33ebbe7055444c1f

Comment by Andrew Morrow (Inactive) [ 14/Aug/20 ]

daniel.moody - I queued this up for v4.4 backport. Feel free to self-approve the backport and take care of it in this sprint.

Comment by Githook User [ 14/Aug/20 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-49457 Ninja tool static links should delete existing archive before linking"
Branch: master
https://github.com/mongodb/mongo/commit/a78139a155bedfd8200e155db66f59837d5796c9

Comment by Daniel Moody [ 05/Aug/20 ]

Ninja ran into this itself when ninja use to use ninja to build itself: https://github.com/ninja-build/ninja/commit/bbc111010163fab044a4782b771fdd6dfa289c66

Seems the recommended solution from ninja is to manually delete the archive yourself.

Comment by Andrew Morrow (Inactive) [ 13/Jul/20 ]

The issue here is extremely subtle. SCons always removes target files before rebuilding them (unless they have been marked .Precious). Ninja on the other hand does not do this.

In a build mode where we produce archive files, an out-of-date archive file, like libtransaction_coordinator.a becomes across this transition, remains on disk as-is when the archiver is re-invoked. Since ar archives are accumulative, this means that the existing symbol information from the prior state is retained.

We will either need to figure out how to convince Ninja to remove archive files before rebuilding them (this may be hard or impossible), or make the archive action in Ninja compound and start with a removal of the archive file (this may also be hard).

This situation only arises when building with --link-model=static and across commits where object files move between archives. The straightorward workaround is to build with --link-model=dynamic.

Comment by Andrew Morrow (Inactive) [ 13/Jul/20 ]

A clean build at 6c0ac0678f3c159c65c497acf155e964dec43f61 doesn't reproduce this issue for me, but that wasn't too surprising. If there is an issue here, it is going to be related to ninja regeneration with respect to the change made for SERVER-49316 in 2a8f91f240f9ec5cb5e47fcbb78a613985de6632.

I checked out the commit before 2a8f91f240f9ec5cb5e47fcbb78a613985de6632 and built, and then checked out 2a8f91f240f9ec5cb5e47fcbb78a613985de6632. And indeed, duplicate symbols appear:

Generating: build.ninja
scons: done building targets.
[10/11] Linking build/opt/mongo/mongod
FAILED: build/opt/mongo/mongod
PATH=/opt/mongodbtoolchain/v3/bin:/usr/local/bin:/opt/bin:/bin:/usr/bin ICECC_CLANG_REMOTE_CPP=1 CCACHE_NOCPP2=1 ICECC_VERSION=/home/andrew/Documents/10gen/dev/src/mongodb/build/scons/icecc/opt_mongodbtoolchain_v3_bin_clang_opt_mongodbtoolchain_v3_bin_clang++.tar.gz CCACHE_PREFIX=/usr/bin/icecc /usr/bin/icerun /opt/mongodbtoolchain/v3/bin/clang++ @build/opt/mongo/mongod.rsp
ld.lld: error: duplicate symbol: mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&)
>>> defined at wait_for_majority_service.cpp:101 (src/mongo/db/s/wait_for_majority_service.cpp:101)
>>>            wait_for_majority_service.o:(mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&)) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
>>> defined at wait_for_majority_service.cpp:101 (src/mongo/db/repl/wait_for_majority_service.cpp:101)
>>>            wait_for_majority_service.o:(.text+0x510) in archive build/opt/mongo/db/repl/libwait_for_majority_service.a
 
ld.lld: error: duplicate symbol: mongo::WaitForMajorityService::_periodicallyWaitForMajority(mongo::ServiceContext*)
>>> defined at wait_for_majority_service.cpp:145 (src/mongo/db/s/wait_for_majority_service.cpp:145)
>>>            wait_for_majority_service.o:(mongo::WaitForMajorityService::_periodicallyWaitForMajority(mongo::ServiceContext*)) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
>>> defined at wait_for_majority_service.cpp:145 (src/mongo/db/repl/wait_for_majority_service.cpp:145)
>>>            wait_for_majority_service.o:(.text+0xA50) in archive build/opt/mongo/db/repl/libwait_for_majority_service.a
 
ld.lld: error: duplicate symbol: mongo::WaitForMajorityService::get(mongo::ServiceContext*)
>>> defined at wait_for_majority_service.cpp:61 (src/mongo/db/s/wait_for_majority_service.cpp:61)
>>>            wait_for_majority_service.o:(mongo::WaitForMajorityService::get(mongo::ServiceContext*)) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
>>> defined at wait_for_majority_service.cpp:61 (src/mongo/db/repl/wait_for_majority_service.cpp:61)
>>>            wait_for_majority_service.o:(.text+0x3C0) in archive build/opt/mongo/db/repl/libwait_for_majority_service.a
 
ld.lld: error: duplicate symbol: mongo::WaitForMajorityService::setUp(mongo::ServiceContext*)
>>> defined at wait_for_majority_service.cpp:65 (src/mongo/db/s/wait_for_majority_service.cpp:65)
>>>            wait_for_majority_service.o:(mongo::WaitForMajorityService::setUp(mongo::ServiceContext*)) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
>>> defined at wait_for_majority_service.cpp:65 (src/mongo/db/repl/wait_for_majority_service.cpp:65)
>>>            wait_for_majority_service.o:(.text+0x3D0) in archive build/opt/mongo/db/repl/libwait_for_majority_service.a
 
ld.lld: error: duplicate symbol: mongo::WaitForMajorityService::shutDown()
>>> defined at wait_for_majority_service.cpp:73 (src/mongo/db/s/wait_for_majority_service.cpp:73)
>>>            wait_for_majority_service.o:(mongo::WaitForMajorityService::shutDown()) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
>>> defined at wait_for_majority_service.cpp:73 (src/mongo/db/repl/wait_for_majority_service.cpp:73)
>>>            wait_for_majority_service.o:(.text+0x130) in archive build/opt/mongo/db/repl/libwait_for_majority_service.a
 
ld.lld: error: duplicate symbol: mongo::WaitForMajorityService::~WaitForMajorityService()
>>> defined at wait_for_majority_service.cpp:57 (src/mongo/db/s/wait_for_majority_service.cpp:57)
>>>            wait_for_majority_service.o:(mongo::WaitForMajorityService::~WaitForMajorityService()) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
>>> defined at wait_for_majority_service.cpp:57 (src/mongo/db/repl/wait_for_majority_service.cpp:57)
>>>            wait_for_majority_service.o:(.text+0x0) in archive build/opt/mongo/db/repl/libwait_for_majority_service.a
 
ld.lld: error: duplicate symbol: mongo::WaitForMajorityService::~WaitForMajorityService()
>>> defined at wait_for_majority_service.cpp:57 (src/mongo/db/s/wait_for_majority_service.cpp:57)
>>>            wait_for_majority_service.o:(mongo::WaitForMajorityService::~WaitForMajorityService()) in archive build/opt/mongo/db/s/libtransaction_coordinator.a
>>> defined at wait_for_majority_service.cpp:57 (src/mongo/db/repl/wait_for_majority_service.cpp:57)
>>>            wait_for_majority_service.o:(.text+0x0) in archive build/opt/mongo/db/repl/libwait_for_majority_service.a
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

Note that build.ninja was in fact regenerated. Since libwait_for_majority_service.a doesn't exist in the commit before, the problem must lie with the build of libtransaction_coordinator.a across the transition.

Generated at Thu Feb 08 05:19:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.