[SERVER-47498] Mongo crash with Got signal: 11 (Segmentation fault) Created: 13/Apr/20  Updated: 27/Oct/23  Resolved: 28/Apr/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.17
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Jean Sumara Assignee: Dmitry Agranat
Resolution: Community Answered Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

We have a problem with mongo 3.6, in a some cases is showing this error and the docker is exiting.

how can I solve this problem?

2020-04-11T23:10:27.888-0400 F - [listener] Invalid access at address: 0x7f08f1b84d13 2020-04-11T23:10:27.972-0400 F - [listener] Got signal: 11 (Segmentation fault).
 0x564bef8c29f1 0x564bef8c1c09 0x564bef8c2276 0x7f095952f390 0x7f0959525e8f 0x564bef77d4db 0x564bef1afa2f 0x564bee287950 0x564bee28815a 0x564bee284f48 0x564bef3dd812 0x564bef3e9ff9 0x564bef3ea241 0x564bef3f448e 0x564bef3db8ce 0x564bef9d1bd0 0x7f09595256ba 0x7f095925b41d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"564BED636000","o":"228C9F1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"564BED636000","o":"228BC09"},{"b":"564BED636000","o":"228C276"},{"b":"7F095951E000","o":"11390"},{"b":"7F095951E000","o":"7E8F","s":"pthread_create"},{"b":"564BED636000","o":"21474DB","s":"_ZN5mongo25launchServiceWorkerThreadESt8functionIFvvEE"},{"b":"564BED636000","o":"1B79A2F","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"564BED636000","o":"C51950","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"564BED636000","o":"C5215A","s":"_ZN5mongo19ServiceStateMachine5startENS0_9OwnershipE"},{"b":"564BED636000","o":"C4EF48","s":"_ZN5mongo21ServiceEntryPointImpl12startSessionESt10shared_ptrINS_9transport7SessionEE"},{"b":"564BED636000","o":"1DA7812"},{"b":"564BED636000","o":"1DB3FF9","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"564BED636000","o":"1DB4241","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"564BED636000","o":"1DBE48E","s":"_ZN4asio10io_context3runEv"},{"b":"564BED636000","o":"1DA58CE"},{"b":"564BED636000","o":"239BBD0"},{"b":"7F095951E000","o":"76BA"},{"b":"7F0959154000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.17", "gitVersion" : "3d6953c361213c5bfab23e51ab274ce592edafe6", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.0-11-amd64", "version" : "#1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11)", "machine" : "x86_64" }, "somap" : [ { "b" : "564BED636000", "elfType" : 3, "buildId" : "E70A565FF1B0F5EB657AB205E08A2E68526CF314" }, { "b" : "7FFF9EB2F000", "elfType" : 3, "buildId" : "E4710F184535B7465A293C8D84196F236A12D2CF" }, { "b" : "7F095A714000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "50A923F8DAFECBCD969C8573116A38C18D0E24D5" }, { "b" : "7F095A2CF000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "15FFEB43278726B025F020862BF51302822A40EC" }, { "b" : "7F095A066000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF69EA60EBE05F2DD689D2B26FC85A73E5FBC3A0" }, { "b" : "7F0959E62000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD" }, { "b" : "7F0959C5A000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "69143E8B39040C964D3958490535322675F15DD3" }, { "b" : "7F0959951000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BAD67A84E56E73D031AE507261DA066B35949D34" }, { "b" : "7F095973B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F095951E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B17C21299099640A6D863E423D99265824E7BB16" }, { "b" : "7F0959154000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "1CA54A6E0D76188105B12E49FE6B8019BF08803A" }, { "b" : "7F095A92F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x564bef8c29f1] mongod(+0x228BC09) [0x564bef8c1c09] mongod(+0x228C276) [0x564bef8c2276] libpthread.so.0(+0x11390) [0x7f095952f390] libpthread.so.0(pthread_create+0x4FF) [0x7f0959525e8f] mongod(_ZN5mongo25launchServiceWorkerThreadESt8functionIFvvEE+0xDB) [0x564bef77d4db] mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x2FF) [0x564bef1afa2f] mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x150) [0x564bee287950] mongod(_ZN5mongo19ServiceStateMachine5startENS0_9OwnershipE+0x13A) [0x564bee28815a] mongod(_ZN5mongo21ServiceEntryPointImpl12startSessionESt10shared_ptrINS_9transport7SessionEE+0xA28) [0x564bee284f48] mongod(+0x1DA7812) [0x564bef3dd812] mongod(_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code+0x389) [0x564bef3e9ff9] mongod(_ZN4asio6detail9scheduler3runERSt10error_code+0xD1) [0x564bef3ea241] mongod(_ZN4asio10io_context3runEv+0x3E) [0x564bef3f448e] mongod(+0x1DA58CE) [0x564bef3db8ce] mongod(+0x239BBD0) [0x564bef9d1bd0] libpthread.so.0(+0x76BA) [0x7f09595256ba] libc.so.6(clone+0x6D) [0x7f095925b41d] ----- END BACKTRACE ----- 2020-04-11T23:10:27.972-0400 I NETWORK [conn2195104] end connection 172.16.0.25:45844 (326 connections now open) 2020-04-11T23:10:27.973-0400 I NETWORK [conn2195107] end connection 172.16.0.25:45854 (325 connections now open)

 



 Comments   
Comment by Dmitry Agranat [ 28/Apr/20 ]

Hi jean.sumara@interlink.com.ar,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Dmitry Agranat [ 16/Apr/20 ]

Thanks jean.sumara@interlink.com.ar,

The max_map_count parameter is another limit that affects the number of connections that can be handled. Each connection requires a thread and each thread requires two mapped memory segments for the stack, so this number needs to be at least twice the number of connections that you want to handle (plus some additional amount to account for memory segments mapped for other purposes, such as the memory allocator heap).

We recommend setting the value for this parameter to 128000.

Please let us know if after setting this value you are still experiencing this issue.

Thanks,
Dima

Comment by Jean Sumara [ 14/Apr/20 ]

Hello Dmitry,

Yes, of course:

root@mongodb:/# cat /proc/sys/vm/max_map_count
65530

Comment by Dmitry Agranat [ 14/Apr/20 ]

Hi jean.sumara@interlink.com.ar, thank you for the report.

Could you please post the output of this command:

cat /proc/sys/vm/max_map_count

Thank you,
Dima

Comment by Jean Sumara [ 13/Apr/20 ]

2020-04-11T23:10:27.972-0400 F -        [listener] Got signal: 11 (Segmentation fault).
 
 0x564bef8c29f1 0x564bef8c1c09 0x564bef8c2276 0x7f095952f390 0x7f0959525e8f 0x564bef77d4db 0x564bef1afa2f 0x564bee287950 0x564bee28815a 0x564bee284f48 0x564bef3dd812 0x564bef3e9ff9 0x564bef3ea241 0x564bef3f448e 0x564bef3db8ce 0x564bef9d1bd0 0x7f09595256ba 0x7f095925b41d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"564BED636000","o":"228C9F1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"564BED636000","o":"228BC09"},{"b":"564BED636000","o":"228C276"},{"b":"7F095951E000","o":"11390"},{"b":"7F095951E000","o":"7E8F","s":"pthread_create"},{"b":"564BED636000","o":"21474DB","s":"_ZN5mongo25launchServiceWorkerThreadESt8functionIFvvEE"},{"b":"564BED636000","o":"1B79A2F","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"564BED636000","o":"C51950","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"564BED636000","o":"C5215A","s":"_ZN5mongo19ServiceStateMachine5startENS0_9OwnershipE"},{"b":"564BED636000","o":"C4EF48","s":"_ZN5mongo21ServiceEntryPointImpl12startSessionESt10shared_ptrINS_9transport7SessionEE"},{"b":"564BED636000","o":"1DA7812"},{"b":"564BED636000","o":"1DB3FF9","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"564BED636000","o":"1DB4241","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"564BED636000","o":"1DBE48E","s":"_ZN4asio10io_context3runEv"},{"b":"564BED636000","o":"1DA58CE"},{"b":"564BED636000","o":"239BBD0"},{"b":"7F095951E000","o":"76BA"},{"b":"7F0959154000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.17", "gitVersion" : "3d6953c361213c5bfab23e51ab274ce592edafe6", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.0-11-amd64", "version" : "#1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11)", "machine" : "x86_64" }, "somap" : [ { "b" : "564BED636000", "elfType" : 3, "buildId" : "E70A565FF1B0F5EB657AB205E08A2E68526CF314" }, { "b" : "7FFF9EB2F000", "elfType" : 3, "buildId" : "E4710F184535B7465A293C8D84196F236A12D2CF" }, { "b" : "7F095A714000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "50A923F8DAFECBCD969C8573116A38C18D0E24D5" }, { "b" : "7F095A2CF000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "15FFEB43278726B025F020862BF51302822A40EC" }, { "b" : "7F095A066000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF69EA60EBE05F2DD689D2B26FC85A73E5FBC3A0" }, { "b" : "7F0959E62000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD" }, { "b" : "7F0959C5A000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "69143E8B39040C964D3958490535322675F15DD3" }, { "b" : "7F0959951000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BAD67A84E56E73D031AE507261DA066B35949D34" }, { "b" : "7F095973B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F095951E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B17C21299099640A6D863E423D99265824E7BB16" }, { "b" : "7F0959154000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "1CA54A6E0D76188105B12E49FE6B8019BF08803A" }, { "b" : "7F095A92F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x564bef8c29f1]
 mongod(+0x228BC09) [0x564bef8c1c09]
 mongod(+0x228C276) [0x564bef8c2276]
 libpthread.so.0(+0x11390) [0x7f095952f390]
 libpthread.so.0(pthread_create+0x4FF) [0x7f0959525e8f]
 mongod(_ZN5mongo25launchServiceWorkerThreadESt8functionIFvvEE+0xDB) [0x564bef77d4db]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x2FF) [0x564bef1afa2f]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x150) [0x564bee287950]
 mongod(_ZN5mongo19ServiceStateMachine5startENS0_9OwnershipE+0x13A) [0x564bee28815a]
 mongod(_ZN5mongo21ServiceEntryPointImpl12startSessionESt10shared_ptrINS_9transport7SessionEE+0xA28) [0x564bee284f48]
 mongod(+0x1DA7812) [0x564bef3dd812]
 mongod(_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code+0x389) [0x564bef3e9ff9]
 mongod(_ZN4asio6detail9scheduler3runERSt10error_code+0xD1) [0x564bef3ea241]
 mongod(_ZN4asio10io_context3runEv+0x3E) [0x564bef3f448e]
 mongod(+0x1DA58CE) [0x564bef3db8ce]
 mongod(+0x239BBD0) [0x564bef9d1bd0]
 libpthread.so.0(+0x76BA) [0x7f09595256ba]
 libc.so.6(clone+0x6D) [0x7f095925b41d]
-----  END BACKTRACE  -----
2020-04-11T23:10:27.972-0400 I NETWORK  [conn2195104] end connection 172.16.0.25:45844 (326 connections now open)
2020-04-11T23:10:27.973-0400 I NETWORK  [conn2195107] end connection 172.16.0.25:45854 (325 connections now open)

Generated at Thu Feb 08 05:14:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.