[SERVER-56214] mongod built from branch master on ARM64 throws "Illegal instruction" Created: 20/Apr/21  Updated: 27/Oct/23  Resolved: 22/Apr/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Manoj Iyer Assignee: Andrew Morrow (Inactive)
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File mongo-debug.png     PNG File mongod-lldb-gcc10.3.png    
Sprint: Dev Platform 2021-05-03
Participants:

 Description   

I built mongod from the latest master branch off of the git repo on ARM64. Ubuntu 20.04 

$ uname -a
Linux arminst8x-01 5.4.0-54-generic #60-Ubuntu SMP Fri Nov 6 10:42:16 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

 

$ git clone https://github.com/mongodb/mongo.git

 

$ buildscripts/scons.py CCFLAGS="-march=armv8.1-a" DESTDIR=/home/ubuntu/bin/mongo --disable-warnings-as-errors install-mongod

 

$ ./mongod --help
Illegal instruction

 

$ lldb ./mongod
(lldb) target create "./mongod"
Current executable set to '/home/ubuntu/bin/mongo/bin/mongod' (aarch64).
(lldb) run
Process 90916 launched: '/home/ubuntu/bin/mongo/bin/mongod' (aarch64)
Process 90916 stopped

  • thread #1, name = 'mongod', stop reason = signal SIGILL: illegal instruction
    frame #0: 0x0000aaaaae759260 mongod`tcmalloc::ThreadCache::InitModule() [inlined] SpinLockHolder::SpinLockHolder(l=<unavailable>, this=<unavailable>) at atomicops-internals-gcc.h:84:30
    81 Atomic32 old_value,
    82 Atomic32 new_value) { 83 Atomic32 prev_value = old_value; -> 84 __atomic_compare_exchange_n(ptr, &prev_value, new_value, 85 0, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED); 86 return prev_value; 87 }

    (lldb) c
    Process 90916 resuming
    Process 90916 exited with status = 4 (0x00000004)
    (lldb) 

 

Please let me know if you need me to attach any logs and where to find them on the build tree. 



 Comments   
Comment by Manoj Iyer [ 22/Apr/21 ]

I built mongod with gcc g++ 10.3 with the default flags you mentioned and I was able to get it to work on an ARM64 system. 

 

$ ./mongod --version
db version v5.0.0-alpha-240-ge9279fb
Build Info: {
"version": "5.0.0-alpha-240-ge9279fb",
"gitVersion": "e9279fb3d14f203842c8dc1e348c73cdc7eb10c5",
"openSSLVersion": "OpenSSL 1.1.1f 31 Mar 2020",
"modules": [],
"allocator": "tcmalloc",
"environment":

{ "distarch": "aarch64", "target_arch": "aarch64" }

}

 

Thanks for your help I think we can close this bug. 

Comment by Andrew Morrow (Inactive) [ 21/Apr/21 ]

Finally, please note that it is important that you pass the -march flag via CCFLAGS and not CFLAGS - they do different things.

Comment by Andrew Morrow (Inactive) [ 21/Apr/21 ]

Sorry, you are right, I meant CCFLAGS="-march=armv8-a" (I forgot the - character in there previously). Still , I really suggest you try that. These errors suggest that your local hardware is not of a sufficiently new processor revision to actually execute armv8.1-a or armv8.2-a instructions. Dropping down to targeting armv8-a will almost certainly get you a working build. Note that on master we build with -moutline-atomics by default if the compiler supports it (I think this is what you are referring to about GCC 10). So by building with CCFLAGS="-march=armv8-a you will get a build that can run on either an armv8-a or an armv8.1-a system and will select the LSE intrinsics if available on the local hardware. Note that you may need to also build with --use-hardware-crc32=off if your local hardware doesn't offer the CRC instructions. For armv8.1-a systems they are I believe mandatory, but they may or may not exist on armv8-a processors.

Comment by Manoj Iyer [ 21/Apr/21 ]

Andrew,

I rebuilt mongod with the default (armv8.2-a) with the 10.3 compiler since the gcc-10 supposedly has a few improvements over atomics. Also CFLAGS="-march=armv8a" is not a valid option.

$ gcc --version
gcc (Ubuntu 10.3.0-1ubuntu1~20.04~2) 10.3.0

 

$ buildscripts/scons.py DESTDIR=/home/ubuntu/bin/mongo --disable-warnings-as-errors install-mongod

 

$ ./mongod --help
Illegal instruction

 

$ lldb ./mongod
(lldb) target create "./mongod"
Current executable set to '/home/ubuntu/bin/mongo/bin/mongod' (aarch64).
(lldb) run
Process 108543 launched: '/home/ubuntu/bin/mongo/bin/mongod' (aarch64)
Process 108543 stopped

  • thread #1, name = 'mongod', stop reason = signal SIGILL: illegal instruction
    frame #0: 0x0000aaaaae735090 mongod`tcmalloc::ThreadCache::InitModule() [inlined] SpinLockHolder::SpinLockHolder(l=<unavailable>, this=0x7f7f7f7f7f7f7f7f) at atomicops-internals-gcc.h:84:30
    81 Atomic32 old_value,
    82 Atomic32 new_value)
    Unknown macro: { 83 Atomic32 prev_value = old_value; -> 84 __atomic_compare_exchange_n(ptr, &prev_value, new_value, 85 0, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED); 86 return prev_value; 87 }(lldb)
Comment by Andrew Morrow (Inactive) [ 21/Apr/21 ]

onlinemails@manjo.net - I note that you have elected to target armv8.1-a here, presumably to step down from targeting of armv8.2-a which is the current default on master. However, are you sure that your local hardware is actually armv8.1-a based? The illegal instruction failure mode suggests that it may not be. If you step down further to target just armv8a with CCFLAGS="-march=armv8a" do things start working again?

Generated at Thu Feb 08 05:38:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.