[SERVER-27380] thin_archive.py scons Tool interferes with SCons error reporting Created: 12/Dec/16  Updated: 06/Dec/17  Resolved: 25/May/17

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 3.5.8

Type: Bug Priority: Major - P3
Reporter: Andrew Morrow (Inactive) Assignee: Andrew Morrow (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-29220 Deploy wholearchive everywhere Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Platforms 2017-05-29
Participants:

 Description   

If the ThinArchive tool is enabled, and you add a non-existent library to LIBDEPS, or create a library dependency cycle, you get an error like this:

scons: *** [build/optdebug/mongo/mongo] TypeError : __class__ assignment: 'Entry' object layout differs from 'ThinArchiveNode'

Instead of the expected

scons: *** [build/optdebug/mongo/libbase.so] Implicit dependency `src/mongo/libasdf.so' not found, needed by target `build/optdebug/mongo/libbase.so'.
scons: done building targets (errors occurred during build).

That happens when the thin_archive.py tool hasn't been loaded.



 Comments   
Comment by Githook User [ 25/May/17 ]

Author:

{u'username': u'acmorrow', u'name': u'Andrew Morrow', u'email': u'acm@mongodb.com'}

Message: SERVER-27380 Revert "Revert "SERVER-27380 Re-enable the thin archive tool""

This reverts commit b4e14a64d7f51846e7c7ed94047ac545e26dce25.
Branch: master
https://github.com/mongodb/mongo/commit/41046adf0e53a35f7c2301f12d577476086c6c6d

Comment by Githook User [ 25/May/17 ]

Author:

{u'username': u'acmorrow', u'name': u'Andrew Morrow', u'email': u'acm@mongodb.com'}

Message: SERVER-27380 Revert "Revert "SERVER-27380 Re-enable the thin archive tool""

This reverts commit b4e14a64d7f51846e7c7ed94047ac545e26dce25.
Branch: master
https://github.com/mongodb/mongo/commit/41046adf0e53a35f7c2301f12d577476086c6c6d

Comment by Andrew Morrow (Inactive) [ 25/May/17 ]

max.hirschhorn - Thanks for the quick revert. My Windows patch queue for this passed, but it required updated images - I suspect the windows build just ran on a lingering non-updated image. The solaris issue though was unexpected from my testing. I'll give it a look.

Comment by Max Hirschhorn [ 25/May/17 ]

Reverted the changes from 3e1461b since compile started to fail on Windows and on Solaris.

Comment by Githook User [ 25/May/17 ]

Author:

{u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}

Message: Revert "SERVER-27380 Re-enable the thin archive tool"

This reverts commit 3e1461b80c42deda1f6b9478e2e574c6d88052f2.
Branch: master
https://github.com/mongodb/mongo/commit/b4e14a64d7f51846e7c7ed94047ac545e26dce25

Comment by Githook User [ 25/May/17 ]

Author:

{u'username': u'acmorrow', u'name': u'Andrew Morrow', u'email': u'acm@mongodb.com'}

Message: SERVER-27380 Re-enable the thin archive tool

This refactors the thin_archive tool to use emitters and scanners
to note that when linking to a thin archive, you must also depend on
the children of that thin archive. Failing to do so is an error,
because a changed .o does not necessarily lead to a different .a,
which would subvert the SCons dependency mechanism.

This also includes a refactoring of the ABILINK tool to use a similar
mechanism, to achieve the opposite effect. For ABILINK, we want to
depend not on the actual .so, but on the hash of its abidw result. We
use emitters, actions, and scanners to produce an associated .abidw
file for each .so we build, and then update the scanner to depend on
the .abidw of our libraries, not the library itself. This allows us to
elide needless relinks.
Branch: master
https://github.com/mongodb/mongo/commit/3e1461b80c42deda1f6b9478e2e574c6d88052f2

Comment by Andrew Morrow (Inactive) [ 28/Feb/17 ]

We have disabled the thin archive tool for now, in the above commit, so this is no longer a pressing issue. We should still fix the Tool, but it is no longer a requirement for 3.5.

Comment by Githook User [ 28/Feb/17 ]

Author:

{u'username': u'acmorrow', u'name': u'Andrew Morrow', u'email': u'acm@mongodb.com'}

Message: SERVER-27380 Use response files for long lines, not thin_archive
Branch: master
https://github.com/mongodb/mongo/commit/27026169e795dc090e692fb27f93645e6646be48

Comment by Gabriel Russell (Inactive) [ 23/Feb/17 ]

There are several instances of the cast in FS.py, but the one we hit is:

https://github.com/mongodb/mongo/blob/master/src/third_party/scons-2.5.0/scons-local-2.5.0/SCons/Node/FS.py#L1069

Comment by Andrew Morrow (Inactive) [ 22/Feb/17 ]

gabriel.russell and jesse - Can you point to where in the SCons sources (preferably the ones embedded in the mongo sources) this cast is done? I'd like to understand a bit better what the problem is before I start thinking about solutions.

Comment by Gabriel Russell (Inactive) [ 21/Feb/17 ]

With some help from jesse, I was able to get to the bottom of this problem. Python doesn't have casting, but there are ad hoc cast implementations that can be made to work in some situations, with some caveats. SCons does use one of these in some places, SCons.Node.FS., in particular. Within this cast implementation it's only possible to cast between direct sub classes of SCons.Node.Fs, which I think are just Entry, File and Dir. Subclasses of these subclasses cause the casting mechanism to fail outright. Because it breaks the casting, it doesn't look like it's OK to subclass SCons.Node.Fs., which is exactly what ThinArchiveNode does.

I was previously thinking that we could just use an actual Scons.Node.Fs.File and override its nested class NodeInfo, but now I realize that just stores "signature" information, and doesn't effect how it's calculated. So I don't think we can use regular Scons.Node.Fs.File nodes for our thin archive nodes.

Subclassing Scons.Node.Node does seam to be supported. There are examples of doing so, SCons/Node/Alias.py and SCons/Node/Python.py. It appears to me that subclassing Scons.Node.Node is the preferred way to create nonstandard scons nodes.

Comment by Andrew Morrow (Inactive) [ 15/Feb/17 ]

And I'll bet that the abilink tool only exhibits it in --link-model=dynamic mode.

Comment by Gabriel Russell (Inactive) [ 15/Feb/17 ]

--link-model=object, --link-model=dynamic, also worked fine, but --link-model=static failed as this ticket explains.

Comment by Andrew Morrow (Inactive) [ 15/Feb/17 ]

Can you test this in all three linking modes (--link-model=object, --link-model=static, --link-model=dynamic). The issue may only affect some of those modes now that SERVER-27577 is fixed.

Comment by Gabriel Russell (Inactive) [ 15/Feb/17 ]

I can't replicate this. With thin_archive.py being loaded, both non existent and circular LIBDEPS dependencies appear to cause correct errors.

This patch causing a circular dependency:

-env.Library("murmurhash3", ["MurmurHash3.cpp"])
+env.Library(
+    target="murmurhash3",
+    source=["MurmurHash3.cpp"],
+    LIBDEPS=[ '$BUILD_DIR/mongo/db/repl/bgsync' ],
+)

compiled:

python ./buildscripts/scons.py -j$(grep -c ^processor /proc/cpuinfo) CC=/opt/mongodbtoolchain/v2/bin/gcc CXX=/opt/mongodbtoolchain/v2/bin/g++ OBJCOPY=/opt/mongodbtoolchain/v2/bin/objcopy --link-model=dynamic

give:

scons: *** [build/opt/mongo/mongod] Library dependency cycle detected: build/opt/mongo/libbase.so => build/opt/third_party/murmurhash3/libmurmurhash3.so => build/opt/mongo/db/repl/libbgsync.so => build/opt/mongo/db/repl/libdata_replicator_external_state_impl.so => build/opt/mongo/db/repl/liboptime.so => build/opt/mongo/libbase.so
scons: building terminated because of errors.
build/opt/mongo/mongod failed: Library dependency cycle detected: build/opt/mongo/libbase.so => build/opt/third_party/murmurhash3/libmurmurhash3.so => build/opt/mongo/db/repl/libbgsync.so => build/opt/mongo/db/repl/libdata_replicator_external_state_impl.so => build/opt/mongo/db/repl/liboptime.so => build/opt/mongo/libbase.so

and with this patch causing a missing dependency:

-env.Library("murmurhash3", ["MurmurHash3.cpp"])
+env.Library(
+    target="murmurhash3",
+    source=["MurmurHash3.cpp"],
+    LIBDEPS=[ '$BUILD_DIR/mofongo' ],
+)

compiled similarly, gives:

scons: *** [build/opt/third_party/murmurhash3/libmurmurhash3.so] Implicit dependency `src/libmofongo.so' not found, needed by target `build/opt/third_party/murmurhash3/libmurmurhash3.so'.
scons: building terminated because of errors.
build/opt/third_party/murmurhash3/libmurmurhash3.so failed: Implicit dependency `src/libmofongo.so' not found, needed by target `build/opt/third_party/murmurhash3/libmurmurhash3.so'.

Comment by Andrew Morrow (Inactive) [ 04/Jan/17 ]

This also applies to the abilink.py tool. The issue should be fixed in both.

Generated at Thu Feb 08 04:14:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.