[SERVER-32773] mlock: Operation not permitted Created: 18/Jan/18  Updated: 13/Aug/19  Resolved: 25/Jan/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kai Brobeil Assignee: Spencer Jackson
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File diagnostic.data.tar.gz    
Issue Links:
Related
related to SERVER-29086 Mongod crashed on mlock Closed
related to SERVER-32915 Set CAP_IPC_LOCK capability in Linux ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

Use mongodb 3.6.1 with the official arch package and try to 'createUser' anywhere.

Participants:

 Description   

When using the command 'createUser', the Server crashes without applying the changes. Please see the log below

 
 
2018-01-15T08:05:07.848+0100 I NETWORK  [conn2] received client metadata from 127.0.0.1:41318 conn: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.6.1" }, os: { type: "Linux", name: "Arch", architecture: "x86_64", version: "rolling" } }
2018-01-15T08:06:14.359+0100 F -        [conn2] Failed to mlock: Operation not permitted
2018-01-15T08:06:14.360+0100 F -        [conn2] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 246
2018-01-15T08:06:14.360+0100 F -        [conn2] 
 
***aborting after fassert() failure
 
 
2018-01-15T08:06:14.515+0100 F -        [conn2] Got signal: 6 (Aborted).
 
 0x55c7479768da 0x55c747975b9e 0x55c747976072 0x7ff3ce3ffdb0 0x7ff3ce06b860 0x7ff3ce06cec9 0x55c7462a88f7 0x55c74784b122 0x55c747848534 0x55c74784890c 0x55c7464d3d26 0x55c747482fb6 0x55c74748103f 0x55c746421dd1 0x55c746423352 0x55c74642586b 0x55c746426a49 0x55c746432dea 0x55c74642e59f 0x55c7464317fc 0x55c7473684a2 0x55c74642d2a4 0x55c74642f4e2 0x55c746430131 0x55c74642e625 0x55c7464317fc 0x55c747368993 0x55c74788f964 0x7ff3ce3f508c 0x7ff3ce12ce1f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55C745A52000","o":"1F248DA","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55C745A52000","o":"1F23B9E"},{"b":"55C745A52000","o":"1F24072"},{"b":"7FF3CE3EE000","o":"11DB0"},{"b":"7FF3CE037000","o":"34860","s":"gsignal"},{"b":"7FF3CE037000","o":"35EC9","s":"abort"},{"b":"55C745A52000","o":"8568F7","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55C745A52000","o":"1DF9122","s":"_ZN5mongo24secure_allocator_details8allocateEmm"},{"b":"55C745A52000","o":"1DF6534","s":"_ZN5mongo5scram15generateSecretsERKNS0_15SCRAMPresecretsE"},{"b":"55C745A52000","o":"1DF690C","s":"_ZN5mongo5scram19generateCredentialsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi"},{"b":"55C745A52000","o":"A81D26","s":"_ZN5mongo13CmdCreateUser3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"55C745A52000","o":"1A30FB6","s":"_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"55C745A52000","o":"1A2F03F","s":"_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"55C745A52000","o":"9CFDD1"},{"b":"55C745A52000","o":"9D1352"},{"b":"55C745A52000","o":"9D386B"},{"b":"55C745A52000","o":"9D4A49","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55C745A52000","o":"9E0DEA","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55C745A52000","o":"9DC59F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55C745A52000","o":"9DF7FC"},{"b":"55C745A52000","o":"19164A2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE"},{"b":"55C745A52000","o":"9DB2A4","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE"},{"b":"55C745A52000","o":"9DD4E2","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55C745A52000","o":"9DE131","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55C745A52000","o":"9DC625","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55C745A52000","o":"9DF7FC"},{"b":"55C745A52000","o":"1916993"},{"b":"55C745A52000","o":"1E3D964"},{"b":"7FF3CE3EE000","o":"708C"},{"b":"7FF3CE037000","o":"F5E1F","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.1", "gitVersion" : "025d4f4fe61efd1fb6f0005be20cb45a004093d1", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.14.3-1-ARCH", "version" : "#1 SMP PREEMPT Thu Nov 30 18:33:13 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55C745A52000", "elfType" : 3, "buildId" : "029D000C7CD46EBE64E61D75949D0CCA9AFE8548" }, { "b" : "7FFE4B36F000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "1B7E6412419608724DC993F8F8D5041F13599D93" }, { "b" : "7FF3D0B90000", "path" : "/usr/lib/libwiredtiger-3.0.0.so", "elfType" : 3, "buildId" : "AF35154F05948BA92AEFEC42B9B37264437FAA8E" }, { "b" : "7FF3D0973000", "path" : "/usr/lib/libsasl2.so.3", "elfType" : 3, "buildId" : "FD2BC2C7EB41B83A66FF6D099037532027A47916" }, { "b" : "7FF3D0718000", "path" : "/usr/lib/libstemmer.so.0", "elfType" : 3, "buildId" : "AD50FFCB248B0279A46FA04967455B1EFD942AF0" }, { "b" : "7FF3D0501000", "path" : "/usr/lib/libz.so.1", "elfType" : 3, "buildId" : "9D65B31D958D72054843331A9D549B1D12ADA045" }, { "b" : "7FF3D02F9000", "path" : "/usr/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "4945D214906C5C184B044DF00ED23A3A4DC08166" }, { "b" : "7FF3D007C000", "path" : "/usr/lib/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "4CA43F102CA009469006E0FA5A595D110E72A88A" }, { "b" : "7FF3CFE09000", "path" : "/usr/lib/libpcre.so.1", "elfType" : 3, "buildId" : "00A48F31C879241B1621FA0EE8A77E24333D006B" }, { "b" : "7FF3CFC00000", "path" : "/usr/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "A97FFE1C498EB6B2E383ABADF091EAF02D7FCA18" }, { "b" : "7FF3CF9E9000", "path" : "/usr/lib/libresolv.so.2", "elfType" : 3, "buildId" : "C37EEAC29624BF0F52FA85E0C5035B0ABAEDC100" }, { "b" : "7FF3CF77F000", "path" : "/usr/lib/libssl.so.1.1", "elfType" : 3, "buildId" : "8B3582F677DE36D14D30BECE4A92F1CD19026299" }, { "b" : "7FF3CF302000", "path" : "/usr/lib/libcrypto.so.1.1", "elfType" : 3, "buildId" : "05A9A6C52AACA0043A1D4A046193015D972AC092" }, { "b" : "7FF3CF0FA000", "path" : "/usr/lib/librt.so.1", "elfType" : 3, "buildId" : "D520E36F95020BC09653CBFA3E3533E2F823713C" }, { "b" : "7FF3CEEF6000", "path" : "/usr/lib/libdl.so.2", "elfType" : 3, "buildId" : "F4C88B82DD709ACA91EB652FFDBD91900B8A908D" }, { "b" : "7FF3CEB6F000", "path" : "/usr/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "14EC457A4EF13B537BA15A26301F1512CE3277F5" }, { "b" : "7FF3CE823000", "path" : "/usr/lib/libm.so.6", "elfType" : 3, "buildId" : "FA35622098EC404C4787159F0AB09FB2F6C5466F" }, { "b" : "7FF3CE60C000", "path" : "/usr/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "BC76DEC7E4673A3AF72A6B26BBB7916359B9802F" }, { "b" : "7FF3CE3EE000", "path" : "/usr/lib/libpthread.so.0", "elfType" : 3, "buildId" : "F2D89412406AED9373964B49A96BA88BAA53A0BC" }, { "b" : "7FF3CE037000", "path" : "/usr/lib/libc.so.6", "elfType" : 3, "buildId" : "48CE2E5F2FF41F8C7911E882A033C8C222B7F209" }, { "b" : "7FF3CDC42000", "path" : "/usr/lib/libtcmalloc.so.4", "elfType" : 3, "buildId" : "475D699EF3D447B31EBBC2ED5B7EEE7702C85447" }, { "b" : "7FF3CDA2D000", "path" : "/usr/lib/liblz4.so.1", "elfType" : 3, "buildId" : "A274C0E4DBC85A4C859DDFFE638FBB9AE9A156AC" }, { "b" : "7FF3D0F05000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E372512D034C6ABD6C233F9CCE96597653F372D3" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x55c7479768da]
 mongod(+0x1F23B9E) [0x55c747975b9e]
 mongod(+0x1F24072) [0x55c747976072]
 libpthread.so.0(+0x11DB0) [0x7ff3ce3ffdb0]
 libc.so.6(gsignal+0x110) [0x7ff3ce06b860]
 libc.so.6(abort+0x1C9) [0x7ff3ce06cec9]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55c7462a88f7]
 mongod(_ZN5mongo24secure_allocator_details8allocateEmm+0x5D2) [0x55c74784b122]
 mongod(_ZN5mongo5scram15generateSecretsERKNS0_15SCRAMPresecretsE+0x94) [0x55c747848534]
 mongod(_ZN5mongo5scram19generateCredentialsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi+0x15C) [0x55c74784890c]
 mongod(_ZN5mongo13CmdCreateUser3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE+0x9D6) [0x55c7464d3d26]
 mongod(_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x76) [0x55c747482fb6]
 mongod(_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x1F) [0x55c74748103f]
 mongod(+0x9CFDD1) [0x55c746421dd1]
 mongod(+0x9D1352) [0x55c746423352]
 mongod(+0x9D386B) [0x55c74642586b]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x539) [0x55c746426a49]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55c746432dea]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x8F) [0x55c74642e59f]
 mongod(+0x9DF7FC) [0x55c7464317fc]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE+0x192) [0x55c7473684a2]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE+0x104) [0x55c74642d2a4]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x6C2) [0x55c74642f4e2]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x1F1) [0x55c746430131]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x115) [0x55c74642e625]
 mongod(+0x9DF7FC) [0x55c7464317fc]
 mongod(+0x1916993) [0x55c747368993]
 mongod(+0x1E3D964) [0x55c74788f964]
 libpthread.so.0(+0x708C) [0x7ff3ce3f508c]
 libc.so.6(clone+0x3F) [0x7ff3ce12ce1f]
-----  END BACKTRACE  -----



 Comments   
Comment by James Harvey [ 13/Aug/19 ]

Would there be a problem with having mongoDB detect a mlock() failure, and gracefully continuing rather than crashing?  Perhaps giving a message saying performance might be improved by making it allowed, like how mongoDB gives a message about numactl?

I'm sure there are some customers who wouldn't want it using mlock() for the potential negative performance impact it can have in certain situations.

The reason I'm asking is as the maintainer of the mongoDB package in the Arch Linux AUR repository.  A lot of our users build packages in a clean chroot environment, using systemd-nspawn.  Prior to 4.2.0, only 8 unit tests crashed due to mlock() so I was able to disable those when inside systemd-nspawn.  So, I could allow the other unit tests, db tests, and integration tests to run.  As of 4.2.0, it looks like mlock() usage has been greatly expanded to the point where it's no longer feasible to find and disable all of the tests crashing from mlock().  After finding and disabling 19 more unit tests, it's appearing likely to me there are now hundreds that would need to be disabled, not only unit tests but also db and integration tests.  The only reasonable choice I have is to disable testing entirely when building inside systemd-nspawn, which I'd like to see fixed.

I realize arguments to systemd-nspawn have been discussed here to allow mlock() usage, and although that could be a solution for users running through systemd-nspawn themselves, I think it's highly unlikely Arch would likely change its clean chroot build scripts (called devtools) to allow mlock() calls, when AFAIK this hasn't been ran into with any other packages.

Comment by Spencer Jackson [ 25/Jan/18 ]

That's great, I'm so glad it worked! Thanks for filing this, you had a fun issue, and I learned a bunch about capabilities.

I'll close out this ticket, so have a great day, and thanks for using MongoDB!

Comment by Kai Brobeil [ 25/Jan/18 ]

You're my hero! It works!

Thank you for your swift and competent analysis. That's definitely another very good reason to use MongoDB for our team (not that I need another one)

I don't see a button to close the issue, I'm guessing you'll do that?

Comment by Spencer Jackson [ 24/Jan/18 ]

OK! I think that's it! Check the man page for systemd-nspawn: https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html There's a flag called --capability, I think you have to bring up your container passing it the CAP_IPC_LOCK capability. The idea is, some syscalls would affect stuff outside the container, so somehow nspawn prevents syscalls which require non-whitelisted capabilities from running. It would be sad if you ran some untrusted code in a container, and it changed your system time to 2038 then mlocked all of your RAM.

There's also a flag called --system-call-filter, which looks like it prevents a bunch of named syscalls from getting run, for pretty much the same reason. A glance at the nspawn source, suggests that it automatically unblacklists mlock if the CAP_IPC_LOCK capability is passed to --capability. So, that's why you're getting "Operation not permitted", even though the soft RLIMIT_MEMLOCK was non-zero: the syscall was just banned. And when you directly set the capability on the binary, the capability itself was banned, so the binary couldn't run.

In short, if you can get nspawn to accept something like --capability=CAP_IPC_LOCK, I think it should work. Your syntax may vary, I've never really used nspawn before.

Comment by Kai Brobeil [ 24/Jan/18 ]

Oh, now that you ask, we're running in an systemd nspawn-container

vault does not work either

/usr/bin/ping does work

maybe after all this is related to nspawn? I wish I understood this more deeply...

Comment by Spencer Jackson [ 24/Jan/18 ]

That's... weird.

When you get "bash: /usr/bin/mongod: Operation not permitted", does anything appear in dmesg, or in journalctl -ef?

Are you using Docker, or any other containerization solution, by any chance? I found a few occurrences of that message in the context of containers.

There appears to be a package called "vault" in the Community repository. If you install it, can the vault binary run? Its PKGBUILD gives it the cap_ipc_lock+ep capability too.

Does /usr/bin/ping work? It uses a different capability.

Comment by Kai Brobeil [ 24/Jan/18 ]

Spencer Jackson, thanks for the explanation.

uname -a: Linux <hostname> 4.14.3-1-ARCH #1 SMP PREEMPT Thu Nov 30 18:33:13 UTC 2017 x86_64 GNU/Linux
getcap: no output
yes it does also happen when starting from the console, usually I use the systemd start script provided by the package
no, no security systems
sure. After doing setcap CAP_IPC_LOCK+ep /usr/bin/mongod, the mongo binary won't start anymore, saying "bash: /usr/bin/mongod: Operation not permitted". Sorry, I don't yet understand this, probably I'm missing something.

Hope this helps

Comment by Spencer Jackson [ 23/Jan/18 ]

yorrd, your error is unexpected. I'm on Arch Linux too, and cannot reproduce this with the mongodb-3.6.1-2 package. "Operation not permitted" means that mlock received an EPERM, which on modern kernels gets emitted when the process doesn't have CAP_IPC_LOCK, and the soft RLIMIT_MEMLOCK limit is 0. If the soft limit is too small, but not 0, I would expect you to get "Cannot allocate memory". Your limits look more than sufficient though.

Can you show me the output of uname -a?
Can you show me the output of getcap /usr/bin/mongod?
How are you starting mongod? systemctl? Does it happen if you run mongod by hand in a terminal?
Do you have any security systems enabled that might not be on a stock installation?
As an experiment, would you be willing to run setcap CAP_IPC_LOCK+ep /usr/bin/mongod on your mongod binary? If you run the command on the globally visible binary, setcap may need to run as root. Alternatively, you may copy the binary to a user controlled directory, and setcap and run the local copy. I believe a binary with this capability should never be able to encounter the error you're observing.

Comment by Kai Brobeil [ 20/Jan/18 ]

sure, here is the ulimit output. As per the other thread, I have increased the memlock to 2147483648 from 16777216, which should be sufficient I figure.

[]# cat /proc/696/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             15308                15308                processes 
Max open files            1024                 4096                 files     
Max locked memory         2147483648           2147483648           bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       15308                15308                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

here are the complete logs.

2018-01-15T08:04:48.905+0100 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-01-15T08:04:48.916+0100 I CONTROL  [initandlisten] MongoDB starting : pid=28535 port=27017 dbpath=/var/lib/mongodb 64-bit host=adornisventures
2018-01-15T08:04:48.916+0100 I CONTROL  [initandlisten] db version v3.6.1
2018-01-15T08:04:48.916+0100 I CONTROL  [initandlisten] git version: 025d4f4fe61efd1fb6f0005be20cb45a004093d1
2018-01-15T08:04:48.916+0100 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0g  2 Nov 2017
2018-01-15T08:04:48.917+0100 I CONTROL  [initandlisten] allocator: tcmalloc
2018-01-15T08:04:48.917+0100 I CONTROL  [initandlisten] modules: none
2018-01-15T08:04:48.917+0100 I CONTROL  [initandlisten] build environment:
2018-01-15T08:04:48.917+0100 I CONTROL  [initandlisten]     distarch: x86_64
2018-01-15T08:04:48.917+0100 I CONTROL  [initandlisten]     target_arch: x86_64
2018-01-15T08:04:48.917+0100 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/var/lib/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2018-01-15T08:04:48.917+0100 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-01-15T08:04:48.917+0100 I STORAGE  [initandlisten] 
2018-01-15T08:04:48.917+0100 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-01-15T08:04:48.917+0100 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-01-15T08:04:48.917+0100 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1416M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-01-15T08:04:49.172+0100 I STORAGE  [initandlisten] WiredTiger message [1515999889:172506][28535:0x7ff3d1101a80], txn-recover: Main recovery loop: starting at 33/5248
2018-01-15T08:04:49.278+0100 I STORAGE  [initandlisten] WiredTiger message [1515999889:278489][28535:0x7ff3d1101a80], txn-recover: Recovering log 33 through 34
2018-01-15T08:04:49.342+0100 I STORAGE  [initandlisten] WiredTiger message [1515999889:342918][28535:0x7ff3d1101a80], txn-recover: Recovering log 34 through 34
2018-01-15T08:04:49.466+0100 I CONTROL  [initandlisten] 
2018-01-15T08:04:49.466+0100 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-01-15T08:04:49.466+0100 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-01-15T08:04:49.466+0100 I CONTROL  [initandlisten] 
2018-01-15T08:04:49.466+0100 I CONTROL  [initandlisten] 
2018-01-15T08:04:49.466+0100 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-01-15T08:04:49.466+0100 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-01-15T08:04:49.467+0100 I CONTROL  [initandlisten] 
2018-01-15T08:04:49.565+0100 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
2018-01-15T08:04:49.566+0100 I NETWORK  [initandlisten] waiting for connections on port 27017
2018-01-15T08:05:07.848+0100 I NETWORK  [conn2] received client metadata from 127.0.0.1:41318 conn: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.6.1" }, os: { type: "Linux", name: "Arch", architecture: "x86_64", version: "rolling" } }
2018-01-15T08:06:14.359+0100 F -        [conn2] Failed to mlock: Operation not permitted
2018-01-15T08:06:14.360+0100 F -        [conn2] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 246
2018-01-15T08:06:14.360+0100 F -        [conn2] 
 
***aborting after fassert() failure
 
 
2018-01-15T08:06:14.515+0100 F -        [conn2] Got signal: 6 (Aborted).
 
 0x55c7479768da 0x55c747975b9e 0x55c747976072 0x7ff3ce3ffdb0 0x7ff3ce06b860 0x7ff3ce06cec9 0x55c7462a88f7 0x55c74784b122 0x55c747848534 0x55c74784890c 0x55c7464d3d26 0x55c747482fb6 0x55c74748103f 0x55c746421dd1 0x55c746423352 0x55c74642586b 0x55c746426a49 0x55c746432dea 0x55c74642e59f 0x55c7464317fc 0x55c7473684a2 0x55c74642d2a4 0x55c74642f4e2 0x55c746430131 0x55c74642e625 0x55c7464317fc 0x55c747368993 0x55c74788f964 0x7ff3ce3f508c 0x7ff3ce12ce1f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55C745A52000","o":"1F248DA","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55C745A52000","o":"1F23B9E"},{"b":"55C745A52000","o":"1F24072"},{"b":"7FF3CE3EE000","o":"11DB0"},{"b":"7FF3CE037000","o":"34860","s":"gsignal"},{"b":"7FF3CE037000","o":"35EC9","s":"abort"},{"b":"55C745A52000","o":"8568F7","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55C745A52000","o":"1DF9122","s":"_ZN5mongo24secure_allocator_details8allocateEmm"},{"b":"55C745A52000","o":"1DF6534","s":"_ZN5mongo5scram15generateSecretsERKNS0_15SCRAMPresecretsE"},{"b":"55C745A52000","o":"1DF690C","s":"_ZN5mongo5scram19generateCredentialsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi"},{"b":"55C745A52000","o":"A81D26","s":"_ZN5mongo13CmdCreateUser3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"55C745A52000","o":"1A30FB6","s":"_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"55C745A52000","o":"1A2F03F","s":"_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"55C745A52000","o":"9CFDD1"},{"b":"55C745A52000","o":"9D1352"},{"b":"55C745A52000","o":"9D386B"},{"b":"55C745A52000","o":"9D4A49","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55C745A52000","o":"9E0DEA","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55C745A52000","o":"9DC59F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55C745A52000","o":"9DF7FC"},{"b":"55C745A52000","o":"19164A2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE"},{"b":"55C745A52000","o":"9DB2A4","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE"},{"b":"55C745A52000","o":"9DD4E2","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55C745A52000","o":"9DE131","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55C745A52000","o":"9DC625","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55C745A52000","o":"9DF7FC"},{"b":"55C745A52000","o":"1916993"},{"b":"55C745A52000","o":"1E3D964"},{"b":"7FF3CE3EE000","o":"708C"},{"b":"7FF3CE037000","o":"F5E1F","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.1", "gitVersion" : "025d4f4fe61efd1fb6f0005be20cb45a004093d1", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.14.3-1-ARCH", "version" : "#1 SMP PREEMPT Thu Nov 30 18:33:13 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55C745A52000", "elfType" : 3, "buildId" : "029D000C7CD46EBE64E61D75949D0CCA9AFE8548" }, { "b" : "7FFE4B36F000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "1B7E6412419608724DC993F8F8D5041F13599D93" }, { "b" : "7FF3D0B90000", "path" : "/usr/lib/libwiredtiger-3.0.0.so", "elfType" : 3, "buildId" : "AF35154F05948BA92AEFEC42B9B37264437FAA8E" }, { "b" : "7FF3D0973000", "path" : "/usr/lib/libsasl2.so.3", "elfType" : 3, "buildId" : "FD2BC2C7EB41B83A66FF6D099037532027A47916" }, { "b" : "7FF3D0718000", "path" : "/usr/lib/libstemmer.so.0", "elfType" : 3, "buildId" : "AD50FFCB248B0279A46FA04967455B1EFD942AF0" }, { "b" : "7FF3D0501000", "path" : "/usr/lib/libz.so.1", "elfType" : 3, "buildId" : "9D65B31D958D72054843331A9D549B1D12ADA045" }, { "b" : "7FF3D02F9000", "path" : "/usr/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "4945D214906C5C184B044DF00ED23A3A4DC08166" }, { "b" : "7FF3D007C000", "path" : "/usr/lib/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "4CA43F102CA009469006E0FA5A595D110E72A88A" }, { "b" : "7FF3CFE09000", "path" : "/usr/lib/libpcre.so.1", "elfType" : 3, "buildId" : "00A48F31C879241B1621FA0EE8A77E24333D006B" }, { "b" : "7FF3CFC00000", "path" : "/usr/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "A97FFE1C498EB6B2E383ABADF091EAF02D7FCA18" }, { "b" : "7FF3CF9E9000", "path" : "/usr/lib/libresolv.so.2", "elfType" : 3, "buildId" : "C37EEAC29624BF0F52FA85E0C5035B0ABAEDC100" }, { "b" : "7FF3CF77F000", "path" : "/usr/lib/libssl.so.1.1", "elfType" : 3, "buildId" : "8B3582F677DE36D14D30BECE4A92F1CD19026299" }, { "b" : "7FF3CF302000", "path" : "/usr/lib/libcrypto.so.1.1", "elfType" : 3, "buildId" : "05A9A6C52AACA0043A1D4A046193015D972AC092" }, { "b" : "7FF3CF0FA000", "path" : "/usr/lib/librt.so.1", "elfType" : 3, "buildId" : "D520E36F95020BC09653CBFA3E3533E2F823713C" }, { "b" : "7FF3CEEF6000", "path" : "/usr/lib/libdl.so.2", "elfType" : 3, "buildId" : "F4C88B82DD709ACA91EB652FFDBD91900B8A908D" }, { "b" : "7FF3CEB6F000", "path" : "/usr/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "14EC457A4EF13B537BA15A26301F1512CE3277F5" }, { "b" : "7FF3CE823000", "path" : "/usr/lib/libm.so.6", "elfType" : 3, "buildId" : "FA35622098EC404C4787159F0AB09FB2F6C5466F" }, { "b" : "7FF3CE60C000", "path" : "/usr/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "BC76DEC7E4673A3AF72A6B26BBB7916359B9802F" }, { "b" : "7FF3CE3EE000", "path" : "/usr/lib/libpthread.so.0", "elfType" : 3, "buildId" : "F2D89412406AED9373964B49A96BA88BAA53A0BC" }, { "b" : "7FF3CE037000", "path" : "/usr/lib/libc.so.6", "elfType" : 3, "buildId" : "48CE2E5F2FF41F8C7911E882A033C8C222B7F209" }, { "b" : "7FF3CDC42000", "path" : "/usr/lib/libtcmalloc.so.4", "elfType" : 3, "buildId" : "475D699EF3D447B31EBBC2ED5B7EEE7702C85447" }, { "b" : "7FF3CDA2D000", "path" : "/usr/lib/liblz4.so.1", "elfType" : 3, "buildId" : "A274C0E4DBC85A4C859DDFFE638FBB9AE9A156AC" }, { "b" : "7FF3D0F05000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E372512D034C6ABD6C233F9CCE96597653F372D3" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x55c7479768da]
 mongod(+0x1F23B9E) [0x55c747975b9e]
 mongod(+0x1F24072) [0x55c747976072]
 libpthread.so.0(+0x11DB0) [0x7ff3ce3ffdb0]
 libc.so.6(gsignal+0x110) [0x7ff3ce06b860]
 libc.so.6(abort+0x1C9) [0x7ff3ce06cec9]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55c7462a88f7]
 mongod(_ZN5mongo24secure_allocator_details8allocateEmm+0x5D2) [0x55c74784b122]
 mongod(_ZN5mongo5scram15generateSecretsERKNS0_15SCRAMPresecretsE+0x94) [0x55c747848534]
 mongod(_ZN5mongo5scram19generateCredentialsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi+0x15C) [0x55c74784890c]
 mongod(_ZN5mongo13CmdCreateUser3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE+0x9D6) [0x55c7464d3d26]
 mongod(_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x76) [0x55c747482fb6]
 mongod(_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x1F) [0x55c74748103f]
 mongod(+0x9CFDD1) [0x55c746421dd1]
 mongod(+0x9D1352) [0x55c746423352]
 mongod(+0x9D386B) [0x55c74642586b]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x539) [0x55c746426a49]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55c746432dea]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x8F) [0x55c74642e59f]
 mongod(+0x9DF7FC) [0x55c7464317fc]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE+0x192) [0x55c7473684a2]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE+0x104) [0x55c74642d2a4]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x6C2) [0x55c74642f4e2]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x1F1) [0x55c746430131]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x115) [0x55c74642e625]
 mongod(+0x9DF7FC) [0x55c7464317fc]
 mongod(+0x1916993) [0x55c747368993]
 mongod(+0x1E3D964) [0x55c74788f964]
 libpthread.so.0(+0x708C) [0x7ff3ce3f508c]
 libc.so.6(clone+0x3F) [0x7ff3ce12ce1f]

Comment by Kelsey Schubert [ 19/Jan/18 ]

Hi yorrd,

Thanks for opening a new ticket. After reviewing the diagnostic.data and stack trace, I have a few more questions to help us with our investigation.

  1. Would you please provide the complete logs from startup to this crash?
  2. Would you please the output of ulimit -a as user the running mongod? For example, execute

    cat /proc/$(pidof mongod)/limits
    

Thank you for your help,
Kelsey

Comment by Kai Brobeil [ 18/Jan/18 ]

diagnostics archive is being attached.

Generated at Thu Feb 08 04:31:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.