[SERVER-31834] Server assert failed when using $where Created: 06/Nov/17  Updated: 07/Dec/17  Resolved: 14/Nov/17

Status: Closed
Project: Core Server
Component/s: JavaScript
Affects Version/s: 3.4.9, 3.4.10
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: wener Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

db.col.find({$where:'true'})

Participants:

 Description   

Server running in docker on alpine host.

Assertion failure: addr == p, at src/third_party/mozjs-45/extract/js/src/jit/ProcessExecutableMemory.cpp:302
2017-11-06T03:19:54.240+0000 F -        [js] Invalid access at address: 0
2017-11-06T03:19:54.270+0000 F -        [js] Got signal: 11 (Segmentation fault).
 
 0x945815af531 0x945815ae749 0x945815aedb6 0x9458190e5f1 0x75f872c3e890 0x945818ce1a4 0x945819dbf99 0x9458198f3ad 0x945819d95ca 0x94581b48ddf 0x94581a00b53 0x94581b8e74c 0x94581ddef35 0x94581b8e935 0x9458196765c 0x9458196781f 0x94581cdef0b 0x94581ce592f 0x94581ce5ab1 0x94581b8d431 0x94581b8d854 0x94581d1f148 0x94581b8c55b 0x945814cd008 0x945814cd747 0x945814fb6d5 0x945814b0bec 0x94582023ac0 0x75f872c37064 0x75f87296c62d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"94580041000","o":"156E531","s":"_ZN5mongo15printStackTraceERSo"},{"b":"94580041000","o":"156D749"},{"b":"94580041000","o":"156DDB6"},{"b":"94580041000","o":"18CD5F1"},{"b":"75F872C2F000","o":"F890"},{"b":"94580041000","o":"188D1A4","s":"_ZN23ProcessExecutableMemory8allocateEmN2js3jit17ProtectionSettingE"},{"b":"94580041000","o":"199AF99","s":"_ZN2js3jit19ExecutableAllocator11systemAllocEm"},{"b":"94580041000","o":"194E3AD","s":"_ZN2js3jit19ExecutableAllocator10createPoolEm"},{"b":"94580041000","o":"19985CA","s":"_ZN2js3jit6Linker7newCodeILNS_7AllowGCE0EEEPNS0_7JitCodeEP9JSContextNS0_8CodeKindE"},{"b":"94580041000","o":"1B07DDF","s":"_ZN2js3jit10JitRuntime33generateProfilerExitFrameTailStubEP9JSContext"},{"b":"94580041000","o":"19BFB53","s":"_ZN2js3jit10JitRuntime10initializeEP9JSContext"},{"b":"94580041000","o":"1B4D74C","s":"_ZN9JSRuntime16createJitRuntimeEP9JSContext"},{"b":"94580041000","o":"1D9DF35","s":"_ZN2JS4Zone13createJitZoneEP9JSContext"},{"b":"94580041000","o":"1B4D935","s":"_ZN13JSCompartment26ensureJitCompartmentExistsEP9JSContext"},{"b":"94580041000","o":"192665C"},{"b":"94580041000","o":"192681F","s":"_ZN2js3jit22CanEnterBaselineMethodEP9JSContextRNS_8RunStateE"},{"b":"94580041000","o":"1C9DF0B","s":"_ZN2js9RunScriptEP9JSContextRNS_8RunStateE"},{"b":"94580041000","o":"1CA492F","s":"_ZN2js13ExecuteKernelEP9JSContextN2JS6HandleIP8JSScriptEER8JSObjectRKNS2_5ValueENS_11ExecuteTypeENS_16AbstractFramePtrEPS9_"},{"b":"94580041000","o":"1CA4AB1","s":"_ZN2js7ExecuteEP9JSContextN2JS6HandleIP8JSScriptEER8JSObjectPNS2_5ValueE"},{"b":"94580041000","o":"1B4C431"},{"b":"94580041000","o":"1B4C854","s":"_ZN2JS8EvaluateEP9JSContextRKNS_22ReadOnlyCompileOptionsEPKcmNS_13MutableHandleINS_5ValueEEE"},{"b":"94580041000","o":"1CDE148","s":"_ZN9JSRuntime15initSelfHostingEP9JSContext"},{"b":"94580041000","o":"1B4B55B","s":"_ZN2js10NewContextEP9JSRuntimem"},{"b":"94580041000","o":"148C008","s":"_ZN5mongo5mozjs14MozJSImplScope10MozRuntimeC2EPKNS0_17MozJSScriptEngineE"},{"b":"94580041000","o":"148C747","s":"_ZN5mongo5mozjs14MozJSImplScopeC1EPNS0_17MozJSScriptEngineE"},{"b":"94580041000","o":"14BA6D5","s":"_ZN5mongo5mozjs15MozJSProxyScope10implThreadEPv"},{"b":"94580041000","o":"146FBEC","s":"_ZN4nspr6Thread13ThreadRoutineEPv"},{"b":"94580041000","o":"1FE2AC0"},{"b":"75F872C2F000","o":"8064"},{"b":"75F872884000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.9", "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.32-0-hardened", "version" : "#1-Alpine SMP Fri Jun 16 12:20:58 GMT 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "94580041000", "elfType" : 3, "buildId" : "153A3B740D234DE361E85D03DF618AFC54076FDC" }, { "b" : "75F873FEA000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "513F73840A2726A043F976949267A14A5A8529B3" }, { "b" : "75F873B6B000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "21115992A1F885E1ACE88AADA60F126AD9759D03" }, { "b" : "75F87376F000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "32E9A5B9EED626E93DEEB00A49033F78652DB9A3" }, { "b" : "75F873567000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "75F873363000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "75F873062000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "75F872E4C000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "75F872C2F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "75F872884000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "75F873DCC000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x945815af531]
 mongod(+0x156D749) [0x945815ae749]
 mongod(+0x156DDB6) [0x945815aedb6]
 mongod(+0x18CD5F1) [0x9458190e5f1]
 libpthread.so.0(+0xF890) [0x75f872c3e890]
 mongod(_ZN23ProcessExecutableMemory8allocateEmN2js3jit17ProtectionSettingE+0x1E4) [0x945818ce1a4]
 mongod(_ZN2js3jit19ExecutableAllocator11systemAllocEm+0x19) [0x945819dbf99]
 mongod(_ZN2js3jit19ExecutableAllocator10createPoolEm+0x4D) [0x9458198f3ad]
 mongod(_ZN2js3jit6Linker7newCodeILNS_7AllowGCE0EEEPNS0_7JitCodeEP9JSContextNS0_8CodeKindE+0x27A) [0x945819d95ca]
 mongod(_ZN2js3jit10JitRuntime33generateProfilerExitFrameTailStubEP9JSContext+0x13CF) [0x94581b48ddf]
 mongod(_ZN2js3jit10JitRuntime10initializeEP9JSContext+0x133) [0x94581a00b53]
 mongod(_ZN9JSRuntime16createJitRuntimeEP9JSContext+0x8C) [0x94581b8e74c]
 mongod(_ZN2JS4Zone13createJitZoneEP9JSContext+0xB5) [0x94581ddef35]
 mongod(_ZN13JSCompartment26ensureJitCompartmentExistsEP9JSContext+0xF5) [0x94581b8e935]
 mongod(+0x192665C) [0x9458196765c]
 mongod(_ZN2js3jit22CanEnterBaselineMethodEP9JSContextRNS_8RunStateE+0x5F) [0x9458196781f]
 mongod(_ZN2js9RunScriptEP9JSContextRNS_8RunStateE+0x18B) [0x94581cdef0b]
 mongod(_ZN2js13ExecuteKernelEP9JSContextN2JS6HandleIP8JSScriptEER8JSObjectRKNS2_5ValueENS_11ExecuteTypeENS_16AbstractFramePtrEPS9_+0xCF) [0x94581ce592f]
 mongod(_ZN2js7ExecuteEP9JSContextN2JS6HandleIP8JSScriptEER8JSObjectPNS2_5ValueE+0xB1) [0x94581ce5ab1]
 mongod(+0x1B4C431) [0x94581b8d431]
 mongod(_ZN2JS8EvaluateEP9JSContextRKNS_22ReadOnlyCompileOptionsEPKcmNS_13MutableHandleINS_5ValueEEE+0xC4) [0x94581b8d854]
 mongod(_ZN9JSRuntime15initSelfHostingEP9JSContext+0x338) [0x94581d1f148]
 mongod(_ZN2js10NewContextEP9JSRuntimem+0x16B) [0x94581b8c55b]
 mongod(_ZN5mongo5mozjs14MozJSImplScope10MozRuntimeC2EPKNS0_17MozJSScriptEngineE+0x3C8) [0x945814cd008]
 mongod(_ZN5mongo5mozjs14MozJSImplScopeC1EPNS0_17MozJSScriptEngineE+0xA7) [0x945814cd747]
 mongod(_ZN5mongo5mozjs15MozJSProxyScope10implThreadEPv+0x95) [0x945814fb6d5]
 mongod(_ZN4nspr6Thread13ThreadRoutineEPv+0x1C) [0x945814b0bec]
 mongod(+0x1FE2AC0) [0x94582023ac0]
 libpthread.so.0(+0x8064) [0x75f872c37064]
 libc.so.6(clone+0x6D) [0x75f87296c62d]
-----  END BACKTRACE  -----
2017-11-06T03:19:56.004+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=38e876a0c691
2017-11-06T03:19:56.005+0000 I CONTROL  [initandlisten] db version v3.4.9
2017-11-06T03:19:56.005+0000 I CONTROL  [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
2017-11-06T03:19:56.005+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2017-11-06T03:19:56.005+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-11-06T03:19:56.005+0000 I CONTROL  [initandlisten] modules: none
2017-11-06T03:19:56.005+0000 I CONTROL  [initandlisten] build environment:
2017-11-06T03:19:56.005+0000 I CONTROL  [initandlisten]     distmod: debian81
2017-11-06T03:19:56.005+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-11-06T03:19:56.005+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-11-06T03:19:56.005+0000 I CONTROL  [initandlisten] options: { security: { authorization: "enabled" } }
2017-11-06T03:19:56.005+0000 W -        [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2017-11-06T03:19:56.014+0000 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-11-06T03:19:56.014+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2017-11-06T03:19:56.014+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31635M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-11-06T03:19:57.371+0000 W CONTROL  [initandlisten]
2017-11-06T03:19:57.371+0000 W CONTROL  [initandlisten] Failed to probe "/sys/kernel/mm/transparent_hugepage": Permission denied
2017-11-06T03:19:57.371+0000 W CONTROL  [initandlisten]
2017-11-06T03:19:57.371+0000 W CONTROL  [initandlisten] Failed to probe "/sys/kernel/mm/transparent_hugepage": Permission denied
2017-11-06T03:19:57.371+0000 I CONTROL  [initandlisten]
2017-11-06T03:19:57.391+0000 W FTDC     [initandlisten] Error getting directory iterator '/sys/block': Permission denied
2017-11-06T03:19:57.391+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2017-11-06T03:19:57.394+0000 I NETWORK  [thread1] waiting for connections on port 27017
2017-11-06T03:19:58.014+0000 I FTDC     [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK

$ neofetch
       .hddddddddddddddddddddddh.          root@localhost
      :dddddddddddddddddddddddddd:         --------------
     /dddddddddddddddddddddddddddd/        OS: Alpine Linux v3.6 x86_64
    +dddddddddddddddddddddddddddddd+       Host: ProLiant DL360e Gen8
  `sdddddddddddddddddddddddddddddddds`     Kernel: 4.9.32-0-hardened
 `ydddddddddddd++hdddddddddddddddddddy`    Uptime: 11 days, 47 mins
.hddddddddddd+`  `+ddddh:-sdddddddddddh.   Packages: 183
hdddddddddd+`      `+y:    .sddddddddddh   Shell: bash 4.3.48
ddddddddh+`   `//`   `.`     -sddddddddd   Terminal: /dev/pts/0
ddddddh+`   `/hddh/`   `:s-    -sddddddd   CPU: Intel Xeon E5-2450L 0 (32) @ 1.800GHz
ddddh+`   `/+/dddddh/`   `+s-    -sddddd   GPU: Matrox Electronics Systems Ltd. MGA G200EH
ddd+`   `/o` :dddddddh/`   `oy-    .yddd   Memory: 2575MiB / 64295MiB
hdddyo+ohddyosdddddddddho+oydddy++ohdddh
.hddddddddddddddddddddddddddddddddddddh.
 `yddddddddddddddddddddddddddddddddddy`
  `sdddddddddddddddddddddddddddddddds`
    +dddddddddddddddddddddddddddddd+
     /dddddddddddddddddddddddddddd/
      :dddddddddddddddddddddddddd:
       .hddddddddddddddddddddddh.



 Comments   
Comment by wener [ 16/Nov/17 ]

I also create a issues for nodejs https://github.com/nodejs/docker-node/issues/588 if they fixed this, I hope mongo fix this too.

Comment by wener [ 16/Nov/17 ]

I use official mongo docker image, I suppose this is maintained by official team. It's not related to alpine, if other os enabled grsec, enabled MPROTECT, this will happen too.

Comment by Mark Agarunov [ 10/Nov/17 ]

Hello wener,

Thank you for the additional information and I'm glad you've found a solution to this problem. Unfortunately at this time neither Alpine Linux nor the grsec patches are officially supported configurations for MongoDB, so there is no established build official process for them. While I am not too familiar with Alpine linux, they appear to maintain their own package repositories, and likely build the packages for Alpine Linux. I would recommend trying to contact the maintainers of the MongoDB package for Alpine Linux with the information you've found.

As I do not see anything that would indicate a bug in the MongoDB Server, I've closed this ticket.

Thanks,
Mark

Comment by wener [ 08/Nov/17 ]

It's caused by gresec, mongo use jit for js, need exec for mmap, by default, this is not allowed, need add a pax header to disable MPROTECT.

This step should added in mongo's build line.

apt-get install -y --no-install-recommends paxctl
paxctl -mc `which mongod`
paxctl -mc `which mongo`

Comment by wener [ 08/Nov/17 ]

I think this is the same problem as
https://github.com/wekan/wekan/issues/1303
https://github.com/arangodb/arangodb/issues/3463
I trying to build a image with pax fix.

Kelsey T Schubert (JIRA) <jira@mongodb.org>于2017年11月8日周三 上午12:15写道:

Comment by Andrew Morrow (Inactive) [ 07/Nov/17 ]

wener - I suspect that Apline linux is prohibiting some memory management operation that the JIT requires. Can you try running under strace and see if you see any syscalls coming back with EPERM or similar immediately before the crash?

Comment by wener [ 07/Nov/17 ]

After `db.adminCommand(

{ setParameter: 1, disableJavaScriptJIT: true }

)`, no more assert failed.

Comment by Andrew Morrow (Inactive) [ 06/Nov/17 ]

Could you try 3.4.10 with the --disableJavaScriptJIT (https://docs.mongodb.com/v3.4/reference/parameters/#param.disableJavaScriptJIT) flag and see if it still crashes?

Comment by wener [ 06/Nov/17 ]

I also tried 3.4.10, still failed.

Generated at Thu Feb 08 04:28:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.