<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:17:08 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-48443] Compilation errors with GCC and ICECC 1.2+</title>
                <link>https://jira.mongodb.org/browse/SERVER-48443</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;After upgrading to Ubuntu 20.04 looks like ccache is broken. Disabling ccache from scons works.&lt;/p&gt;

&lt;p&gt;The lack of ccache is however a &lt;b&gt;significant&lt;/b&gt; productivity issue.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Ubuntu 20.04&lt;br/&gt;
 ccache version 3.7.7&lt;br/&gt;
 ICECC 1.3.1&lt;/p&gt;</description>
                <environment></environment>
        <key id="1363735">SERVER-48443</key>
            <summary>Compilation errors with GCC and ICECC 1.2+</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="ryan.egesdahl@mongodb.com">Ryan Egesdahl</assignee>
                                    <reporter username="svilen.mihaylov@mongodb.com">Svilen Mihaylov</reporter>
                        <labels>
                    </labels>
                <created>Wed, 27 May 2020 15:32:44 +0000</created>
                <updated>Sun, 29 Oct 2023 22:07:43 +0000</updated>
                            <resolved>Fri, 17 Jul 2020 21:35:33 +0000</resolved>
                                                    <fixVersion>4.4.1</fixVersion>
                    <fixVersion>4.7.0</fixVersion>
                                    <component>Build</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="3323976" author="xgen-internal-githook" created="Thu, 6 Aug 2020 04:15:30 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Ryan Egesdahl&apos;, &apos;email&apos;: &apos;ryan.egesdahl@mongodb.com&apos;, &apos;username&apos;: &apos;deriamis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48443&quot; title=&quot;Compilation errors with GCC and ICECC 1.2+&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48443&quot;&gt;&lt;del&gt;SERVER-48443&lt;/del&gt;&lt;/a&gt; Fix builds with Icecream 1.2+ and gcc 4.4+&lt;/p&gt;

&lt;p&gt;A bug spotted in Icecream 1.2+ can cause build failures when building&lt;br/&gt;
with gcc. This is, in turn, due to a bug in GCC where the preprocessor&lt;br/&gt;
executed via `gcc -E` has different behavior than the one used&lt;br/&gt;
internally during compilation. We are working with Icecream, and GCC&lt;br/&gt;
to address these problems. For now, we work around the bugs.&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;GCC bug report: &lt;a href=&quot;https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88475&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88475&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;Icecream bug report: icecc/icecream#550&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;(cherry picked from commit f69c932697877c4036aca066cf212eaad55be070)&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/f3cd65f32a6d05803b3f3a8d2f5d32f6d9e95135&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/f3cd65f32a6d05803b3f3a8d2f5d32f6d9e95135&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3290855" author="xgen-internal-githook" created="Fri, 17 Jul 2020 21:34:06 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Ryan Egesdahl&apos;, &apos;email&apos;: &apos;ryan.egesdahl@mongodb.com&apos;, &apos;username&apos;: &apos;deriamis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48443&quot; title=&quot;Compilation errors with GCC and ICECC 1.2+&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48443&quot;&gt;&lt;del&gt;SERVER-48443&lt;/del&gt;&lt;/a&gt; Fix builds with Icecream 1.2+ and gcc 4.4+&lt;/p&gt;

&lt;p&gt;A bug spotted in Icecream 1.2+ can cause build failures when building&lt;br/&gt;
with gcc. This is, in turn, due to a bug in GCC where the preprocessor&lt;br/&gt;
executed via `gcc -E` has different behavior than the one used&lt;br/&gt;
internally during compilation. We are working with Icecream, and GCC&lt;br/&gt;
to address these problems. For now, we work around the bugs.&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;GCC bug report: &lt;a href=&quot;https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88475&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88475&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;Icecream bug report: icecc/icecream#550&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/f69c932697877c4036aca066cf212eaad55be070&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/f69c932697877c4036aca066cf212eaad55be070&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="3290824" author="xgen-internal-githook" created="Fri, 17 Jul 2020 21:20:34 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Ryan Egesdahl&apos;, &apos;email&apos;: &apos;ryan.egesdahl@mongodb.com&apos;, &apos;username&apos;: &apos;deriamis&apos;}
&lt;p&gt;Message: Revert &quot;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48443&quot; title=&quot;Compilation errors with GCC and ICECC 1.2+&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48443&quot;&gt;&lt;del&gt;SERVER-48443&lt;/del&gt;&lt;/a&gt; Fix builds with Icecream 1.2+ and gcc 4.4+&quot;&lt;/p&gt;

&lt;p&gt;This reverts commit 01dd381f3359c44bbc9338d91371d1ff823bb7d8.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/4635320ea16bb9217605b7150461561fdded5e22&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/4635320ea16bb9217605b7150461561fdded5e22&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3285506" author="svilen.mihaylov" created="Wed, 15 Jul 2020 02:51:02 +0000"  >&lt;p&gt;Thank you very much for fixing this!&lt;/p&gt;</comment>
                            <comment id="3285399" author="xgen-internal-githook" created="Tue, 14 Jul 2020 23:45:48 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Ryan Egesdahl&apos;, &apos;email&apos;: &apos;ryan.egesdahl@mongodb.com&apos;, &apos;username&apos;: &apos;deriamis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48443&quot; title=&quot;Compilation errors with GCC and ICECC 1.2+&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48443&quot;&gt;&lt;del&gt;SERVER-48443&lt;/del&gt;&lt;/a&gt; Fix builds with Icecream 1.2+ and gcc 4.4+&lt;/p&gt;

&lt;p&gt;A bug spotted in Icecream 1.2+ can cause build failures when building&lt;br/&gt;
with gcc. This is, in turn, due to a bug in GCC where the preprocessor&lt;br/&gt;
executed via `gcc -E` has different behavior than the one used&lt;br/&gt;
internally during compilation. We are working with Icecream, and GCC&lt;br/&gt;
to address these problems. For now, we work around the bugs.&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;GCC bug report: &lt;a href=&quot;https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88475&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88475&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;Icecream bug report: &lt;a href=&quot;https://github.com/icecc/icecream/issues/550&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/icecc/icecream/issues/550&lt;/a&gt;&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/01dd381f3359c44bbc9338d91371d1ff823bb7d8&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/01dd381f3359c44bbc9338d91371d1ff823bb7d8&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="3272769" author="ryan.egesdahl" created="Tue, 7 Jul 2020 17:08:45 +0000"  >&lt;p&gt;After many false starts and dead ends, this ticket has become a cautionary tale about not believing the first thing you see. What&apos;s actually happening here has nothing to do with Icecream or ccache. It&apos;s actually a GCC bug:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88475&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88475&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;What&apos;s &lt;b&gt;actually&lt;/b&gt; actually happening here is that we have a header somewhere in our source tree (exactly where is another question at the moment) that triggers the bug. Specifically, when one or more directives is processed by the GCC preprocessor with &lt;tt&gt;-fdirectives-only&lt;/tt&gt;, it gets expanded into a form that is not further parseable by GCC. Clang does not have this problem because the corresponding flag we pass to it does not have the same behavior or the same issues.&lt;/p&gt;

&lt;p&gt;The reason why this issue happens only with ccache and Icecream enabled in our build system is an accident of how we have them configured to work together versus separately. What makes this problem even more complicated to understand is the fact that &lt;b&gt;both&lt;/b&gt; Icecream &lt;b&gt;and&lt;/b&gt; ccache can trigger the behavior, multiple causal factors are in play, the exact relationships between the tools are murky, and the fact that it&apos;s possible to accidentally work around this problem and not see it again until you do another full rebuild.&lt;/p&gt;

&lt;p&gt;In short, passing &lt;tt&gt;-fdirectives-only&lt;/tt&gt; is the real triggering issue in our build system at the moment. I will work around the problem here. Whether and how we correct the source to fix the real cause will be a separate problem.&lt;/p&gt;</comment>
                            <comment id="3226619" author="ryan.egesdahl" created="Fri, 26 Jun 2020 22:36:08 +0000"  >&lt;p&gt;It looks like I was slightly wrong in the above explanation. The cause of breakage is accurate, but the mechanism wasn&apos;t quite so much. The problem was not, in fact, &lt;tt&gt;CCACHE_NOCPP2&lt;/tt&gt; not being set. Rather, it was &lt;tt&gt;-fdirectives-only&lt;/tt&gt; not being set that did it. The &lt;tt&gt;CCACHE_NOCPP2&lt;/tt&gt; environment variable does not actually disable all preprocessing - it just disables a second preprocessor that is used to calculate a hash for the cache.&lt;/p&gt;

&lt;p&gt;What seems to be happening is that ccache is using icecream as its preprocessor, and the newer version of icecream &lt;b&gt;by default&lt;/b&gt; also does remote preprocessing. Because we use &lt;tt&gt;-fdirectives-only&lt;/tt&gt;, this has been triggering a known issue with newer icecream versions: &lt;a href=&quot;https://github.com/icecc/icecream/issues/451&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/icecc/icecream/issues/451&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;There are two mechanisms of failure: first, &lt;tt&gt;-fdirectives-only&lt;/tt&gt; causes problems for ccache when it&apos;s is not running its own preprocessor (and is using icecream instead) because it then tries to get icecream to compile it a second time, which fails for the above reason; second, running a preprocessor &lt;b&gt;without&lt;/b&gt; &lt;tt&gt;-fdirectives-only&lt;/tt&gt; causes ccache to send the preprocessed text it got back from icecream back for compiling, which triggers a second preprocessing step that GCC can&apos;t handle because it is expecting to do something like that via a precompiled header.&lt;/p&gt;

&lt;p&gt;When I added &lt;tt&gt;-fpch-includes&lt;/tt&gt; to the mix, I was actually causing ccache to intercept the compilation failure (because it has special processing for that flag) and try it again from scratch, which succeeded. I verified that by watching ccache errors for precompiled headers ticking upwards even though the compilation was succeeding. Icecream ignores the flag, so it was having no effect remotely. I was still passing in &lt;tt&gt;-fdirectives-only&lt;/tt&gt; at this point, which is why cache was succeeding - the text returned from the remote icecream preprocessing matched what ccache was producing locally.&lt;/p&gt;

&lt;p&gt;The recommended solution from both the Icecream and GCC developers is that we do &lt;b&gt;no&lt;/b&gt; preprocessing remotely (and disable the &lt;tt&gt;-fdirectives-only&lt;/tt&gt; flag). I&apos;ve verified this works as well as doing preprocessing both locally &lt;b&gt;and&lt;/b&gt; remotely &lt;b&gt;with&lt;/b&gt; the flag. I am not sure which one is better for performance, but I lean towards the external recommendation here.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=acm&quot; class=&quot;user-hover&quot; rel=&quot;acm&quot;&gt;acm&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.moody&quot; class=&quot;user-hover&quot; rel=&quot;daniel.moody&quot;&gt;daniel.moody&lt;/a&gt; - thoughts on this?&lt;/p&gt;</comment>
                            <comment id="3225192" author="ryan.egesdahl" created="Fri, 26 Jun 2020 05:35:05 +0000"  >&lt;p&gt;This behavior started in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40943&quot; title=&quot;Support fetching icecream toolchain tarballs from URLs in SCons&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40943&quot;&gt;&lt;del&gt;SERVER-40943&lt;/del&gt;&lt;/a&gt; and was caused by adding ccache as a tool unconditionally. This was causing &lt;tt&gt;icecream.py&lt;/tt&gt; to use ccache but without the &lt;tt&gt;CCACHE_NOCPP2&lt;/tt&gt; option we had set to prevent exactly this problem. I only noticed this during an unrelated SCons-only build where I had icecream enabled and ccache disabled and had turned on VERBOSE=1 to see what was actually being compiled. That&apos;s when I noticed ccache was being used even though I had not requested it.&lt;/p&gt;

&lt;p&gt;This was actually a bad interaction between &lt;tt&gt;ccache&lt;/tt&gt; and &lt;tt&gt;icecc&lt;/tt&gt; preprocessing, which I will get to shortly. As for why we were getting different behavior only on newer versions of icecream, this commit to the icecream repo tells the rest of the story:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/icecc/icecream/commit/e7d95da05ed78a051d45c9cfb00fc9f1539505ca&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/icecc/icecream/commit/e7d95da05ed78a051d45c9cfb00fc9f1539505ca&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Essentially, before icecream 1.2, precompiled headers were not being detected properly anyway and were not being used remotely. This patch was committed in icecream version 1.1, so everyone using that version never noticed that &lt;tt&gt;CCACHE_NO_CPP2&lt;/tt&gt; wasn&apos;t set. Versions 1.0.1 and 1.3, however, had the correct behavior and &lt;tt&gt;CCACHE_NO_CPP2&lt;/tt&gt; mattered again. Interestingly, this commit tells us what my original proposed fix (adding &quot;-fpch-preprocess&quot; to the compiler flags) actually did:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/icecc/icecream/commit/c56c19ea0264ef295a3f5824beb1ada48bd58682&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/icecc/icecream/commit/c56c19ea0264ef295a3f5824beb1ada48bd58682&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The flow of information is that icecream was preprocessing the headers and then returning them to ccache, which would then &lt;b&gt;also&lt;/b&gt; attempt to preprocess the same headers. Clang handles this gracefully, but GCC did not. The flag, which was dropped by icecream (thanks to the above commit) was still being used by ccache, which was allowing the compilation to succeed by telling GCC that a precompiled header was present.&lt;/p&gt;

&lt;p&gt;The mess of logic for enabling ccache and icecream and then detecting whether they were enabled and which flags and environment options were being passed where were confounding a problem that was already difficult to pull apart. I&apos;m going to correct it, simplify it, and document it better so we don&apos;t run into this problem again.&lt;/p&gt;</comment>
                            <comment id="3220628" author="acm" created="Tue, 23 Jun 2020 21:07:25 +0000"  >&lt;p&gt;Notably, this does not happen on Ubuntu 18.04, where icecc is 1.1 and ccache is 3.4.1 On Ubuntu 20.04, we find icecc 1.3 and ccache 3.7.7. That doesn&apos;t really help us rule much out unfortunately, since both are newer versions. But it does narrow the range of versions in play.&lt;/p&gt;</comment>
                            <comment id="3211558" author="ryan.egesdahl" created="Wed, 17 Jun 2020 00:26:40 +0000"  >&lt;p&gt;I think I have finally isolated the problem. What seems to be happening is that the source is getting preprocessed by &lt;b&gt;both&lt;/b&gt; the compiler &lt;b&gt;and&lt;/b&gt; icecream (as documented here: &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Using_Icecream#Frequent_problems_debugging&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Using_Icecream#Frequent_problems_debugging&lt;/a&gt; under &quot;I get build failures to due -Werror&quot;). Clang appears not to care that this has happened, but (newer?) GCC does for some reason. The fix is simple - since we know that using both icecream and ccache together results in a twice-compiled header, we can just apply the fix suggested here: &lt;a href=&quot;https://ccache.dev/manual/3.7.9.html#_precompiled_headers&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://ccache.dev/manual/3.7.9.html#_precompiled_headers&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Doing the above fix seems to correct the problem, and I can verify that both ccache and icecream are in use. I&apos;ll test further and ensure it doesn&apos;t break anything else.&lt;/p&gt;

&lt;p&gt;Conditions to reproduce:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;icecc 1.3.1 (or 1.1+)&lt;/li&gt;
	&lt;li&gt;building with gcc&lt;/li&gt;
	&lt;li&gt;using ccache&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I was able to reproduce this with both Ninja and SCons builds in the above conditions.&lt;/p&gt;</comment>
                            <comment id="3211506" author="ryan.egesdahl" created="Tue, 16 Jun 2020 23:34:48 +0000"  >&lt;p&gt;It turns out that in the previous investigation, I was not actually using icecc at all because of the way I was setting the ICECC variable for SCons. However, I was able to perform several other tests that demonstrate where the problem likely is.&lt;/p&gt;

&lt;p&gt;First, when I disable the ccache cache (using &lt;tt&gt;CCACHE_DISABLE=1&lt;/tt&gt;), the problem does not appear. Second, when I add &lt;tt&gt;CCACHE_PREFIX_CPP=&amp;lt;icecc_path&amp;gt;&lt;/tt&gt; to &lt;tt&gt;icecream.py&lt;/tt&gt;, the source compiles with some warnings that look like this:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Compiling build/debug/mongo/db/pipeline/document_source_change_stream_close_cursor.dyn.o&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;In file included from src/mongo/db/query/explain.h:32,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                 from src/mongo/db/commands.h:47,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                 from build/debug/mongo/db/stats/server_write_concern_metrics_gen.cpp:16:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;src/mongo/db/exec/plan_stage.h: In member function &apos;void mongo::PlanStage::dispose(mongo::OperationContext*)&apos;:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;src/mongo/db/exec/plan_stage.h:296:62: error: self-comparison always evaluates to true [-Werror=tautological-compare]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;             invariant(!_opCtx || opCtx == opCtx);&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                                                              ^&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;cc1plus: all warnings being treated as errors&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;scons: *** [build/debug/mongo/db/stats/server_write_concern_metrics_gen.dyn.o] Error 1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;In file included from src/mongo/db/query/explain.h:32,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                 from src/mongo/db/commands.h:47,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                 from src/mongo/db/commands/server_status.h:32,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                 from src/mongo/db/stats/server_read_concern_metrics.cpp:34:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;src/mongo/db/exec/plan_stage.h: In member function &apos;void mongo::PlanStage::dispose(mongo::OperationContext*)&apos;:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;src/mongo/db/exec/plan_stage.h:296:62: error: self-comparison always evaluates to true [-Werror=tautological-compare]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;             invariant(!_opCtx || opCtx == opCtx);&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This suggests that there is a problem in the handoff between the &quot;local&quot; and &quot;remote&quot; compilers at the preprocessing stage, and specifically that the remote preprocessor is somehow incorrect. The locale therefore seems to be a red herring here because I can verify that the environment passes through as I expect:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ubuntu@ip-10-122-5-51:~/workspace/mongo$ LANG=&quot;blah&quot; ccache icerun sh -c &apos;env&apos; | grep LANG&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;LANG=blah&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ubuntu@ip-10-122-5-51:~/workspace/mongo$ LANG=&quot;blah&quot; ccache icecc sh -c &apos;env&apos; | grep LANG&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;LANG=blah&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;It now looks like this problem might be related to something &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=acm&quot; class=&quot;user-hover&quot; rel=&quot;acm&quot;&gt;acm&lt;/a&gt; is already working on (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47598&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-47598&lt;/a&gt;) where we have noticed some inconsistencies in the compiler that gets packaged for icecream runs. I suspect that fixing the problem there will fix the problem here, but I am going to investigate further to try and confirm that.&lt;/p&gt;</comment>
                            <comment id="3211323" author="ryan.egesdahl" created="Tue, 16 Jun 2020 21:35:52 +0000"  >&lt;p&gt;This seems to be in the interaction between ccache and icecream. Specifically, when ccache is enabled with icecream, we call ccache by setting &lt;tt&gt;CCACHE_PREFIX&lt;/tt&gt; in &lt;tt&gt;icecream.py&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;We probably didn&apos;t see this because it depends on the following factors being present at the same time:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;ccache version &amp;gt; 3.5.1 or 1.6&lt;/li&gt;
	&lt;li&gt;environment has been &quot;sanitized&quot; (usually because SCons does this)&lt;/li&gt;
	&lt;li&gt;ccache is being used with icecream&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The combination of the above three items is unusual in our current environment because it only seems to show up in a specific use case on Ubuntu 20.04 right now. However, it&apos;s worth fixing because it represents behavior we have been depending on which changed out from underneath us.&lt;/p&gt;

&lt;p&gt;My current hypothesis based on some testing is that icecream is being run with a different set of LANG environment variables than gcc is being run under via CCACHE, which may in turn be related to this bug which was fixed in ccache 3.5.1 and 3.6:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/ccache/ccache/issues/327&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/ccache/ccache/issues/327&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If this is what is happening, it may be that this is causing some form of character encoding issues, the exact nature of which I currently do not understand. However, specifically setting LANG when icecream is run seems to &quot;fix&quot; the problem. I&apos;m investigating further in this area.&lt;/p&gt;</comment>
                            <comment id="3210044" author="svilen.mihaylov" created="Tue, 16 Jun 2020 15:24:54 +0000"  >&lt;p&gt;Ok this is great. Its great we are validating gcc vs clang.&lt;/p&gt;</comment>
                            <comment id="3210029" author="acm" created="Tue, 16 Jun 2020 15:21:30 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=svilen.mihaylov&quot; class=&quot;user-hover&quot; rel=&quot;svilen.mihaylov&quot;&gt;svilen.mihaylov&lt;/a&gt; - Our toolchain of record on Linux is GCC, macOS is Xcode clang, and Windows is MSVC. So Linux clang builds aren&apos;t how we produce our official release binaries. However, using clang for development is just fine, and we do have clang based linux builders on the waterfall, notably the &lt;tt&gt;&amp;#42;SAN&lt;/tt&gt; builders. So you should have no issues using the toolchain clang on Linux for your development work.&lt;/p&gt;</comment>
                            <comment id="3209053" author="svilen.mihaylov" created="Mon, 15 Jun 2020 22:16:18 +0000"  >&lt;p&gt;Thank you very much for investigating. I have uploaded my build.ninja file. &lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I tried switching the vars from gcc to clang and it seems to be working!&lt;/p&gt;

&lt;p&gt;Before I am completely satisfied with the resolution, I&apos;d like to know how this affects (if at all) the binaries which are produced? Are we fully cross-compatible?&lt;/p&gt;</comment>
                            <comment id="3208989" author="acm" created="Mon, 15 Jun 2020 21:48:55 +0000"  >&lt;p&gt;I was able to reproduce this on an Ubuntu 20.04 spawn host using the above command line, and then running &lt;tt&gt;ninja build/debug/mongo/base/data_range.dyn.o&lt;/tt&gt; with a clean &lt;tt&gt;ccache&lt;/tt&gt;, so there is definitely some sort of issue here.&lt;/p&gt;

&lt;p&gt;I was able to make the issue go away by enabling only one of icecream or ccache at the same time, so that also helps - the issue is in the integration where we handle the case where both are enabled. However, and notably, the issue did not reproduce if I switched to using toolchain clang by replacing &lt;tt&gt;&amp;#45;&amp;#45;variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars&lt;/tt&gt; with &lt;tt&gt;&amp;#45;&amp;#45;variables-files=etc/scons/mongodbtoolchain_stable_clang.vars&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=svilen.mihaylov&quot; class=&quot;user-hover&quot; rel=&quot;svilen.mihaylov&quot;&gt;svilen.mihaylov&lt;/a&gt; - could you see if switching to clang works for you? In the meanwhile, we will continue to investigate why this seems not to work as expected.&lt;/p&gt;</comment>
                            <comment id="3208475" author="svilen.mihaylov" created="Mon, 15 Jun 2020 17:21:37 +0000"  >&lt;p&gt;Just tried on newest available toolchain version (revisions/4666020dcdac5e2c493adc9e1638877b50ee26c6/) with the same problem.&lt;/p&gt;</comment>
                            <comment id="3208155" author="svilen.mihaylov" created="Mon, 15 Jun 2020 15:21:17 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=acm&quot; class=&quot;user-hover&quot; rel=&quot;acm&quot;&gt;acm&lt;/a&gt; Is there a specific set of info you are interested in terms of environment setup?&lt;/p&gt;</comment>
                            <comment id="3208153" author="svilen.mihaylov" created="Mon, 15 Jun 2020 15:20:44 +0000"  >&lt;p&gt;Attached a text file showing the compile errors when using CCACHE.&lt;/p&gt;</comment>
                            <comment id="3208142" author="svilen.mihaylov" created="Mon, 15 Jun 2020 15:16:05 +0000"  >&lt;p&gt;Sure, here:&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;./buildscripts/scons.py --dbg=on --opt=off --link-model=dynamic --allocator=system CCFLAGS=&apos;-Wa,--compress-debug-sections -gsplit-dwarf&apos; --variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars MONGO_VERSION=$(git describe --abbrev=0 | tail -c+2) ICECC=icecc CCACHE=ccache --ninja --modules=enterprise build.ninja&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="3208063" author="acm" created="Mon, 15 Jun 2020 14:47:28 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=svilen.mihaylov&quot; class=&quot;user-hover&quot; rel=&quot;svilen.mihaylov&quot;&gt;svilen.mihaylov&lt;/a&gt; - Could you add complete repro instructions to this ticket? Local environment info, commands executed, error messages generated, etc. That would go a long way to helping us investigate.&lt;/p&gt;</comment>
                            <comment id="3207957" author="svilen.mihaylov" created="Mon, 15 Jun 2020 14:15:00 +0000"  >&lt;p&gt;Just did a quick re-testing and it is still broken.&lt;/p&gt;</comment>
                            <comment id="3205469" author="svilen.mihaylov" created="Thu, 11 Jun 2020 16:34:06 +0000"  >&lt;p&gt;Can I help with the investigation of this problem?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="266018" name="build.ninja" size="34257339" author="svilen.mihaylov@mongodb.com" created="Mon, 15 Jun 2020 22:16:10 +0000"/>
                            <attachment id="265944" name="error.txt" size="2000213" author="svilen.mihaylov@mongodb.com" created="Mon, 15 Jun 2020 15:19:58 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 15 Jun 2020 14:47:28 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 27 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 27 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andrew.morrow@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>ryan.egesdahl@mongodb.com</customfieldvalue>
            <customfieldvalue>svilen.mihaylov@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxn8i7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxe60v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3947">Dev Platform 2020-06-15</customfieldvalue>
    <customfieldvalue id="4006">Dev Platform 2020-06-29</customfieldvalue>
    <customfieldvalue id="4007">Dev Platform 2020-07-13</customfieldvalue>
    <customfieldvalue id="4088">Dev Platform 2020-07-27</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;A minimal repro for this on an Ubuntu 20.04 spawn host is:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;python3 ./buildscripts/scons.py --dbg=on --opt=off --link-model=dynamic --variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars ICECC=icecc CCACHE=ccache build/debug/mongo/base/data_range.dyn.o&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxmurj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>