[SERVER-11093] Compile failures on MCI Linux 64-bit builders Created: 08/Oct/13  Updated: 11/Jul/16  Resolved: 11/Oct/13

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 2.5.3

Type: Bug Priority: Major - P3
Reporter: Matt Kangas Assignee: Matt Kangas
Resolution: Done Votes: 0
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

Ernie notified us by email at 3:16 PM Tuesday, but it's still unresolved at 7 PM. Ticketing for tracking purposes so we aren't left wondering why MCI Linux compiles are busted.

340c96304d46 Linux 64-bit compile

340c96304d46 Linux 64-bit DEBUG compile

340c96304d46 Linux 64-bit DEBUG DUR OFF compile

340c96304d46 Enterprise Linux 64-bit Amazon AMI compile

[2013/10/08 22:55:48.228] /
[2013/10/08 22:55:48.228] usr/bin/ld: /usr/lib/gcc/x86_64-amazon-linux/4.6.3/libstdc++.a(codecvt.o): relocation R_X86_64_32S against `vtable for std::codecvt<char, char, __mbstate_t>' can not be used when making a shared object; recompile with -fPIC
[2013/10/08 22:55:48.228] /usr/lib/gcc/x86_64-amazon-linux/4.6.3/libstdc++.a: could not read symbols: Bad value
[2013/10/08 22:55:48.242] collect2: ld returned 1 exit status



 Comments   
Comment by Ernie Hershey [ 11/Oct/13 ]

These should all be working fine now.

Comment by Ernie Hershey [ 09/Oct/13 ]

This problem should be gone for all but the debug duroff builds. I just deployed a fix for them that should start in on some new compiles shortly this evening. As soon as compiles are verified on the different build variants with that fix in (after around 6pm tonight), I'll sign back off on linux 64 builders.

Right now all linux-64 builders other than debug duroff should be working.

Comment by Matt Kangas [ 09/Oct/13 ]

Spoke to Ian. Since these issues have been broken out into MCI tickets, let's unassign this ticket.

Ian also has an idea for labeling MCI tickets related to test failures, so we can see those in our queue too. I dig it.

Comment by Eric Milkie [ 09/Oct/13 ]

Ernie said the OS X builds are failing because they are on dedicated hardware and there is a bug in MCI that causes it to run multiple tasks at the same time on a given instance. So you will see random errors like object files getting deleted out from underneath the compiler, for example.

The debug dur off error seems to be an issue with MCI. We can't tell what it was running at the time of the failure. It looks like it should have been running the linker, but the error message claims it was running the binary named 'test'.

Comment by Matt Kangas [ 09/Oct/13 ]

Failing compiles as of rev 606b6ee6ca (10/09/13 4:24am zulu time)

606b6ee6cab9 Enterprise Linux 64-bit Amazon AMI compile

  • as Eric noted, same link error as above

606b6ee6cab9 Linux 64-bit DEBUG DUR OFF compile

[2013/10/09 05:08:41.884] scons: *** [build/linux2/cc__opt_rh_devtoolset-2_root_usr_bin_gcc/cxx__opt_rh_devtoolset-2_root_usr_bin_g++/dbg_on/durableDefaultOff/opt_on/mongo/test] Error 127

  • It looks like this failed twice. Search for the string "scons: building terminated because of errors" in the buildlog. Did MCI deliberately re-run this task?

606b6ee6cab9 OS X 10.8 DEBUG compile
606b6ee6cab9 OS X 10.8 DUR OFF compile

First failure seems to be:

[2013/10/09 01:14:58.193] {standard input}:unknown:FATAL:
[2013/10/09 01:14:58.193] can't create output file: build/darwin/allocator_system/dbg_on/opt_on/third_party/v8/src/x64/builtins-x64.o
[2013/10/09 01:14:58.264] scons: *** [build/darwin/allocator_system/dbg_on/opt_on/third_party/v8/src/x64/builtins-x64.o] Error 1

Comment by Eric Milkie [ 09/Oct/13 ]

Notably, the Enterprise Linux Amazon AMI builder is still suffering from this issue.

Comment by Ernie Hershey [ 09/Oct/13 ]

Compiles should start succeeding again in most cases but I need to fix mongoperf on physical servers and test a bit more before sending the all-clear.

Generated at Thu Feb 08 03:24:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.