[SERVER-33395] PPC64 little endian altivec optimizations are broken on newer gcc Created: 20/Feb/18  Updated: 08/Jan/24  Resolved: 04/May/18

Status: Closed
Project: Core Server
Component/s: Packaging
Affects Version/s: 3.4.13, 3.6.2
Fix Version/s: 3.4.16, 3.6.6, 4.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Marek Skalický Assignee: Mira Carey
Resolution: Fixed Votes: 0
Labels: bkp
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ppc64le-fix-altivec.patch    
Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.6, v3.4
Steps To Reproduce:

$
$ cat variables.list
VERBOSE=1
VARIANT_DIR="fedora"
$ cat build-options
-j4 --use-system-tcmalloc --wiredtiger=on --nostrip --disable-warnings-as-errors --variables-files=variables.list --opt=off
$ scons build/fedora/mongo/db/fts/fts_unicode_phrase_matcher_test build/fedora/mongo/db/fts/unicode/byte_vector_test build/fedora/mongo/db/fts/unicode/string_test $(cat build-options)
$ build/fedora/mongo/db/fts/fts_unicode_phrase_matcher_test; build/fedora/mongo/db/fts/unicode/byte_vector_test; build/fedora/mongo/db/fts/unicode/string_test

Sprint: Platforms 2018-03-26, Platforms 2018-05-07
Participants:

 Description   

Hi,
I'm building mongodb on ppc64le platform and I see error fails related to ByteVector altivec.

$ build/fedora/mongo/db/fts/fts_unicode_phrase_matcher_test; build/fedora/mongo/db/fts/unicode/byte_vector_test; build/fedora/mongo/db/fts/unicode/string_test
2018-02-20T14:16:24.384+0100 I -        [main] going to run suite: FtsUnicodePhraseMatcher
2018-02-20T14:16:24.384+0100 I -        [main] 	 going to run test: CaseAndDiacriticInsensitive
2018-02-20T14:16:24.384+0100 E -        [main] Throwing exception: Expected: phraseMatcher.phraseMatches(find1, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:48
2018-02-20T14:16:24.385+0100 I -        [main] FAIL: CaseAndDiacriticInsensitive	Expected: phraseMatcher.phraseMatches(find1, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:48
2018-02-20T14:16:24.385+0100 I -        [main] 	 going to run test: CaseSensitiveAndDiacriticInsensitive
2018-02-20T14:16:24.385+0100 E -        [main] Throwing exception: Expected: phraseMatcher.phraseMatches(find1, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:68
2018-02-20T14:16:24.385+0100 I -        [main] FAIL: CaseSensitiveAndDiacriticInsensitive	Expected: phraseMatcher.phraseMatches(find1, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:68
2018-02-20T14:16:24.385+0100 I -        [main] 	 going to run test: CaseInsensitiveAndDiacriticSensitive
2018-02-20T14:16:24.385+0100 E -        [main] Throwing exception: Expected: phraseMatcher.phraseMatches(find1, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:88
2018-02-20T14:16:24.385+0100 I -        [main] FAIL: CaseInsensitiveAndDiacriticSensitive	Expected: phraseMatcher.phraseMatches(find1, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:88
2018-02-20T14:16:24.385+0100 I -        [main] 	 going to run test: CaseAndDiacriticSensitive
2018-02-20T14:16:24.385+0100 I -        [main] 	 going to run test: CaseAndDiacriticInsensitiveTurkish
2018-02-20T14:16:24.385+0100 E -        [main] Throwing exception: Expected: phraseMatcher.phraseMatches(find2, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:129
2018-02-20T14:16:24.385+0100 I -        [main] FAIL: CaseAndDiacriticInsensitiveTurkish	Expected: phraseMatcher.phraseMatches(find2, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:129
2018-02-20T14:16:24.385+0100 I -        [main] 	 DONE running tests
2018-02-20T14:16:24.385+0100 I -        [main] **************************************************
2018-02-20T14:16:24.385+0100 I -        [main] FtsUnicodePhraseMatcher        | tests:    5 | fails:    4 | assert calls:          0 | time secs:  0.000
	CaseAndDiacriticInsensitive	Expected: phraseMatcher.phraseMatches(find1, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:48
	CaseSensitiveAndDiacriticInsensitive	Expected: phraseMatcher.phraseMatches(find1, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:68
	CaseInsensitiveAndDiacriticSensitive	Expected: phraseMatcher.phraseMatches(find1, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:88
	CaseAndDiacriticInsensitiveTurkish	Expected: phraseMatcher.phraseMatches(find2, str, options) @src/mongo/db/fts/fts_unicode_phrase_matcher_test.cpp:129
2018-02-20T14:16:24.385+0100 I -        [main] TOTALS                         | tests:    5 | fails:    4 | assert calls:          0 | time secs:  0.000
2018-02-20T14:16:24.385+0100 I -        [main] Failing tests:
2018-02-20T14:16:24.385+0100 I -        [main] 	 FtsUnicodePhraseMatcher/CaseAndDiacriticInsensitive Failed
2018-02-20T14:16:24.385+0100 I -        [main] 	 FtsUnicodePhraseMatcher/CaseSensitiveAndDiacriticInsensitive Failed
2018-02-20T14:16:24.385+0100 I -        [main] 	 FtsUnicodePhraseMatcher/CaseInsensitiveAndDiacriticSensitive Failed
2018-02-20T14:16:24.385+0100 I -        [main] 	 FtsUnicodePhraseMatcher/CaseAndDiacriticInsensitiveTurkish Failed
2018-02-20T14:16:24.385+0100 I -        [main] FAILURE - 4 tests in 1 suites failed
2018-02-20T14:16:24.394+0100 I -        [main] going to run suite: ByteVector
2018-02-20T14:16:24.394+0100 I -        [main] 	 going to run test: LoadStoreUnaligned
2018-02-20T14:16:24.394+0100 I -        [main] 	 going to run test: Splat
2018-02-20T14:16:24.394+0100 I -        [main] 	 going to run test: MaskAny
2018-02-20T14:16:24.394+0100 E -        [main] Throwing exception: Expected ByteVector::countInitialZeros(mask) == offset (16 == 0) @src/mongo/db/fts/unicode/byte_vector_test.cpp:70
2018-02-20T14:16:24.394+0100 I -        [main] FAIL: MaskAny	Expected ByteVector::countInitialZeros(mask) == offset (16 == 0) @src/mongo/db/fts/unicode/byte_vector_test.cpp:70
2018-02-20T14:16:24.394+0100 I -        [main] 	 going to run test: MaskHigh
2018-02-20T14:16:24.394+0100 E -        [main] Throwing exception: Expected ByteVector::countInitialZeros(mask) == offset (16 == 0) @src/mongo/db/fts/unicode/byte_vector_test.cpp:82
2018-02-20T14:16:24.395+0100 I -        [main] FAIL: MaskHigh	Expected ByteVector::countInitialZeros(mask) == offset (16 == 0) @src/mongo/db/fts/unicode/byte_vector_test.cpp:82
2018-02-20T14:16:24.395+0100 I -        [main] 	 going to run test: CompareEQ
2018-02-20T14:16:24.395+0100 I -        [main] 	 going to run test: CompareGT
2018-02-20T14:16:24.395+0100 I -        [main] 	 going to run test: CompareLT
2018-02-20T14:16:24.395+0100 I -        [main] 	 going to run test: BitOr
2018-02-20T14:16:24.395+0100 I -        [main] 	 going to run test: BitOrAssign
2018-02-20T14:16:24.395+0100 I -        [main] 	 going to run test: BitAnd
2018-02-20T14:16:24.395+0100 I -        [main] 	 going to run test: BitAndAssign
2018-02-20T14:16:24.395+0100 I -        [main] 	 DONE running tests
2018-02-20T14:16:24.395+0100 I -        [main] **************************************************
2018-02-20T14:16:24.395+0100 I -        [main] ByteVector                     | tests:   11 | fails:    2 | assert calls:          0 | time secs:  0.000
	MaskAny	Expected ByteVector::countInitialZeros(mask) == offset (16 == 0) @src/mongo/db/fts/unicode/byte_vector_test.cpp:70
	MaskHigh	Expected ByteVector::countInitialZeros(mask) == offset (16 == 0) @src/mongo/db/fts/unicode/byte_vector_test.cpp:82
2018-02-20T14:16:24.395+0100 I -        [main] TOTALS                         | tests:   11 | fails:    2 | assert calls:          0 | time secs:  0.000
2018-02-20T14:16:24.395+0100 I -        [main] Failing tests:
2018-02-20T14:16:24.395+0100 I -        [main] 	 ByteVector/MaskAny Failed
2018-02-20T14:16:24.395+0100 I -        [main] 	 ByteVector/MaskHigh Failed
2018-02-20T14:16:24.395+0100 I -        [main] FAILURE - 2 tests in 1 suites failed
2018-02-20T14:16:24.403+0100 I -        [main] going to run suite: UnicodeString
2018-02-20T14:16:24.403+0100 I -        [main] 	 going to run test: SubstrTest
2018-02-20T14:16:24.403+0100 I -        [main] 	 going to run test: RemoveDiacritics
2018-02-20T14:16:24.404+0100 E -        [main] Throwing exception: Expected output + filler == String::caseFoldAndStripDiacritics(&buf, input + filler, kCaseSensitive, kNormal) (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx == ^xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) @src/mongo/db/fts/unicode/string_test.cpp:88
2018-02-20T14:16:24.404+0100 I -        [main] FAIL: RemoveDiacritics	Expected output + filler == String::caseFoldAndStripDiacritics(&buf, input + filler, kCaseSensitive, kNormal) (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx == ^xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) @src/mongo/db/fts/unicode/string_test.cpp:88
2018-02-20T14:16:24.404+0100 I -        [main] 	 going to run test: CaseFolding
2018-02-20T14:16:24.405+0100 I -        [main] FAIL: CaseFolding	 std::exception: text contains invalid UTF-8 in test CaseFolding
2018-02-20T14:16:24.406+0100 I -        [main] 	 going to run test: CaseFoldingTurkish
2018-02-20T14:16:24.406+0100 E -        [main] Throwing exception: Expected "kac yasındasınız" == String::caseFoldAndStripDiacritics(&buf, test1, kDiacriticSensitive, kTurkish) (kac yasındasınız == kac yasindasiniz) @src/mongo/db/fts/unicode/string_test.cpp:138
2018-02-20T14:16:24.406+0100 I -        [main] FAIL: CaseFoldingTurkish	Expected "kac yasındasınız" == String::caseFoldAndStripDiacritics(&buf, test1, kDiacriticSensitive, kTurkish) (kac yasındasınız == kac yasindasiniz) @src/mongo/db/fts/unicode/string_test.cpp:138
2018-02-20T14:16:24.406+0100 I -        [main] 	 going to run test: CaseFoldingAndRemoveDiacritics
2018-02-20T14:16:24.407+0100 E -        [main] Throwing exception: Expected "ποσο χρονων εισαι?" == String::caseFoldAndStripDiacritics(&buf, test1, 0, kNormal) (ποσο χρονων εισαι? == Πόσο χρονών είσαι?) @src/mongo/db/fts/unicode/string_test.cpp:151
2018-02-20T14:16:24.407+0100 I -        [main] FAIL: CaseFoldingAndRemoveDiacritics	Expected "ποσο χρονων εισαι?" == String::caseFoldAndStripDiacritics(&buf, test1, 0, kNormal) (ποσο χρονων εισαι? == Πόσο χρονών είσαι?) @src/mongo/db/fts/unicode/string_test.cpp:151
2018-02-20T14:16:24.407+0100 I -        [main] 	 going to run test: SubstringMatch
2018-02-20T14:16:24.407+0100 I -        [main] FAIL: SubstringMatch	 std::exception: text contains invalid UTF-8 in test SubstringMatch
2018-02-20T14:16:24.407+0100 I -        [main] 	 going to run test: SubstringMatchTurkish
2018-02-20T14:16:24.407+0100 E -        [main] Throwing exception: Expected: String::substrMatch(str, "yasındasınız", String::kNone, CaseFoldMode::kTurkish) @src/mongo/db/fts/unicode/string_test.cpp:182
2018-02-20T14:16:24.407+0100 I -        [main] FAIL: SubstringMatchTurkish	Expected: String::substrMatch(str, "yasındasınız", String::kNone, CaseFoldMode::kTurkish) @src/mongo/db/fts/unicode/string_test.cpp:182
2018-02-20T14:16:24.408+0100 I -        [main] 	 going to run test: BadUTF8
2018-02-20T14:16:24.408+0100 I -        [main] 	 going to run test: UTF32ToUTF8
2018-02-20T14:16:24.408+0100 I -        [main] 	 DONE running tests
2018-02-20T14:16:24.408+0100 I -        [main] **************************************************
2018-02-20T14:16:24.408+0100 I -        [main] UnicodeString                  | tests:    9 | fails:    6 | assert calls:          0 | time secs:  0.005
	RemoveDiacritics	Expected output + filler == String::caseFoldAndStripDiacritics(&buf, input + filler, kCaseSensitive, kNormal) (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx == ^xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) @src/mongo/db/fts/unicode/string_test.cpp:88
	CaseFolding	 std::exception: text contains invalid UTF-8 in test CaseFolding
	CaseFoldingTurkish	Expected "kac yasındasınız" == String::caseFoldAndStripDiacritics(&buf, test1, kDiacriticSensitive, kTurkish) (kac yasındasınız == kac yasindasiniz) @src/mongo/db/fts/unicode/string_test.cpp:138
	CaseFoldingAndRemoveDiacritics	Expected "ποσο χρονων εισαι?" == String::caseFoldAndStripDiacritics(&buf, test1, 0, kNormal) (ποσο χρονων εισαι? == Πόσο χρονών είσαι?) @src/mongo/db/fts/unicode/string_test.cpp:151
	SubstringMatch	 std::exception: text contains invalid UTF-8 in test SubstringMatch
	SubstringMatchTurkish	Expected: String::substrMatch(str, "yasındasınız", String::kNone, CaseFoldMode::kTurkish) @src/mongo/db/fts/unicode/string_test.cpp:182
2018-02-20T14:16:24.409+0100 I -        [main] TOTALS                         | tests:    9 | fails:    6 | assert calls:          0 | time secs:  0.005
2018-02-20T14:16:24.409+0100 I -        [main] Failing tests:
2018-02-20T14:16:24.409+0100 I -        [main] 	 UnicodeString/RemoveDiacritics Failed
2018-02-20T14:16:24.409+0100 I -        [main] 	 UnicodeString/CaseFolding Failed
2018-02-20T14:16:24.410+0100 I -        [main] 	 UnicodeString/CaseFoldingTurkish Failed
2018-02-20T14:16:24.410+0100 I -        [main] 	 UnicodeString/CaseFoldingAndRemoveDiacritics Failed
2018-02-20T14:16:24.410+0100 I -        [main] 	 UnicodeString/SubstringMatch Failed
2018-02-20T14:16:24.410+0100 I -        [main] 	 UnicodeString/SubstringMatchTurkish Failed
2018-02-20T14:16:24.410+0100 I -        [main] FAILURE - 6 tests in 1 suites failed

When I disable/remove altivec include in src/mongo/db/fts/unicode/byte_vector.h everything works.

When I debuged byte_vector_test and error {{FAIL: MaskAny Expected ByteVector::countInitialZeros(mask) == offset (16 == 0) @src/mongo/db/fts/unicode/byte_vector_test.cpp:70}} I figured out that vec_extract(vec_vbpermq(_data, bits), 0) in ByteVector::maskHight() returns 0.

I don't know if this is mongodb problem or problem in compiler,... Do you have any idea how to easily test it? (I don't know ppc64 altivec instructions well) What could be the minimal reproducer in case of compiler problem?



 Comments   
Comment by Githook User [ 22/May/18 ]

Author:

{'username': 'hanumantmk', 'name': 'Jason Carey', 'email': 'jcarey@argv.me'}

Message: SERVER-33395 Add configure check for vec_vbpermq output

GCC changed the output of vec_vbpermq between 5.4.0 and later. This
seems likely to be a bug, probably in earlier versions of gcc. Rather
than attempt to enumerate how the compiler versions work, this adds a
runtime configure check which will select the lane that works.

(cherry picked from commit b989fe4b0cfba4062c5bb7fb68337f188070c1c8)
Branch: v3.6
https://github.com/mongodb/mongo/commit/97de8e700531bf5e2169ec2a844598a81302167b

Comment by Githook User [ 16/May/18 ]

Author:

{'email': 'jcarey@argv.me', 'username': 'hanumantmk', 'name': 'Jason Carey'}

Message: SERVER-33395 Add configure check for vec_vbpermq output

GCC changed the output of vec_vbpermq between 5.4.0 and later. This
seems likely to be a bug, probably in earlier versions of gcc. Rather
than attempt to enumerate how the compiler versions work, this adds a
runtime configure check which will select the lane that works.

(cherry picked from commit b989fe4b0cfba4062c5bb7fb68337f188070c1c8)
Branch: v3.4
https://github.com/mongodb/mongo/commit/879e111740b0cc69db61ca334031164496b3adc9

Comment by Mira Carey [ 16/May/18 ]

No worries mskalick,

I'm currently planning on making a backport to 3.6, but it probably won't land until 3.6.6 (just missed getting it in for the 3.6.5 release).

Comment by Marek Skalický [ 16/May/18 ]

Hi jason. I'm sorry, but currently I'm not building master branch... so it was easier for me to try to apply the patch to 3.6.4 (see attachment).

And with this patch unittests and core JS tests are passing for me on both Little and Big Endian ppc64.

Comment by Mira Carey [ 04/May/18 ]

mskalick, I've pushed a change to pick up at configure time where vec_vbpermq puts it's output.  If you test against the master branch of mongodb, and it fixes your issue, I'll go ahead with the backports

Comment by Githook User [ 04/May/18 ]

Author:

{'email': 'jcarey@argv.me', 'name': 'Jason Carey', 'username': 'hanumantmk'}

Message: SERVER-33395 Add configure check for vec_vbpermq output

GCC changed the output of vec_vbpermq between 5.4.0 and later. This
seems likely to be a bug, probably in earlier versions of gcc. Rather
than attempt to enumerate how the compiler versions work, this adds a
runtime configure check which will select the lane that works.
Branch: master
https://github.com/mongodb/mongo/commit/b989fe4b0cfba4062c5bb7fb68337f188070c1c8

Comment by Marek Skalický [ 24/Apr/18 ]

Current implementation is working with new gcc on ppc64 Big Endian. And with changes proposed in this ticket, the tests are failing on ppc64be.

Comment by Apollon Oikonomopoulos [ 23/Mar/18 ]

This is causing build failures on Debian as well. Applying the change fixes the issue, so I've included it in the latest upload.

Comment by Dimitri John Ledkov [ 22/Mar/18 ]

I can confirm we are experiencing same test suite failures on Ubuntu now too. Will try out the proposed change.

Comment by Marek Skalický [ 12/Mar/18 ]

Thanks. If you have any question or anything to test, feel free to ask.

Comment by Mira Carey [ 09/Mar/18 ]

mskalick, I wrote the altivec byte vector implementation. It's perfectly possible that I tried the one side of vbpermq's output, failed to see bytes, then tried the other and stuck with what worked (on gcc 5.4). I'll check the docs and see what the api actually specifies. If that fails we've got some contacts at IBM I can follow up with.

Comment by Marek Skalický [ 07/Mar/18 ]

I can confirm that with this change all three mentioned tests pass.

Can you ask someone who wrote this code to know what should be right? If gcc 5.4 has a bug or newer gcc has one.

Comment by Mira Carey [ 06/Mar/18 ]

Hmm, interesting. This code was written for little endian, and continues to work in our CI environment with gcc 5.4 (on that platform)

I suppose the interesting story is if gcc 5.4 had the bug, and later versions actually have the fix.

If you swap the 0 for a 1 in byte_vector

diff --git a/src/mongo/db/fts/unicode/byte_vector_altivec.h b/src/mongo/db/fts/unicode/byte_vector_altivec.h
index f7a07ac074..74a6441c00 100644
--- a/src/mongo/db/fts/unicode/byte_vector_altivec.h
+++ b/src/mongo/db/fts/unicode/byte_vector_altivec.h
@@ -98,7 +98,7 @@ public:
         // big endian by comparison.
         const Native bits = {120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 0};
 
-        return vec_extract(vec_vbpermq(_data, bits), 0);
+        return vec_extract(vec_vbpermq(_data, bits), 1);
     }
 
     /**

Do unit tests pass for you?

If so... I suppose I can add explicit compiler version checks, or look into checking the endianness of vec_vbpermq in our configure step

Comment by Marek Skalický [ 05/Mar/18 ]

Jason, please take a look in the gcc bug report.

It seems that reproduces is really doing what it should do. So Dave is asking if you really want to use uint64_t mask = vec_extract(vec_vbpermq(vec, bits), 0);? With 0 substituted by 1 reproducer is working fine...

Comment by Marek Skalický [ 28/Feb/18 ]

I don't think that this is about optimizations. Even with g++ -O0 your reproducer aborts.

Comment by Mira Carey [ 27/Feb/18 ]

Thanks for filing the bug report and for the link!

Comment by Marek Skalický [ 27/Feb/18 ]

Awesome! Thank you.

It aborts. So I reported a bug - https://bugzilla.redhat.com/show_bug.cgi?id=1549628

Comment by Mira Carey [ 23/Feb/18 ]

I transcribed out what that test does into a minimal repro. It should abort if vbpermq is working differently than expected:

#include <altivec.h>
#include <cstring>
#include <cstdint>
#include <cstdlib>
 
int main() {
    using Native = __vector signed char;
    const size_t size = sizeof(Native);
    const Native bits = {120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 0};
 
    uint8_t inputBuf[size];
    std::memset(inputBuf, 0xFF, sizeof(inputBuf));
 
    for (size_t offset = 0; offset <= size; offset++) {
        Native vec = vec_vsx_ld(0, reinterpret_cast<const Native*>(inputBuf));
        
        uint64_t mask = vec_extract(vec_vbpermq(vec, bits), 0);
        
        size_t initialZeros = (mask == 0 ? size : __builtin_ctzll(mask));
        if (initialZeros != offset) {
            std::abort();
        }
        
        if (offset < size) { 
            inputBuf[offset] = 0;  // Add an initial 0 for the next loop.
        }
    }
}

Comment by Marek Skalický [ 23/Feb/18 ]

I'm not saying it is not compiler bug. Interesting is that it appeared to me in three different environments.
(with gcc 8.0.1 and 7.3.1 - both with fix of bug, you mentioned)

Comment by Mira Carey [ 20/Feb/18 ]

My inclination is to say compiler bug.

For context:
The byte_vector_test for MaskAny doesn't do very much (it uses some altivec magic to pull all the high bits from 16 bytes into a 16 bit output value). It pretty much only tests those two intrinsics.
The primitives it uses are vbpermq (which does the bit extraction via a passed uint array) and vec_extract (vbpermq outputs to two 64 bit output values, one of them has the 16 bits we care about, the other is zeroed).

The test is green in our CI environment today (https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_71_ppc64le_compile_all_2ae87330910dd7af58507c77d0363d267be8381e_18_02_16_21_39_39) with gcc 5.4.0

It appears that there is a current gcc bug live on 5.5.0, 6.4.1, 7.2.1, 8.0 which breaks vbpermq in particular (seems like a bad optimization pass)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84033

Any chance you're on one of those affected versions?

Generated at Thu Feb 08 04:33:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.