[SERVER-21217] mongos crash issue on FreeBSD 10.1-RELEASE Created: 30/Oct/15  Updated: 20/Feb/16  Resolved: 26/Nov/15

Status: Closed
Project: Core Server
Component/s: Networking
Affects Version/s: 3.2.0-rc0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Andrew Shevchuk Assignee: Adam Midvidy
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

uname -a
FreeBSD www.lambda-cloud.net 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64


Attachments: File mongos.log    
Issue Links:
Duplicate
is duplicated by SERVER-22740 mongod crashes on FreeBSD while setin... Closed
Operating System: FreeBSD
Sprint: Platform C (11/20/15), Platform D (12/11/15)
Participants:

 Description   

mongos 3.2.0-rc0 crush on FreeBSD 10.1-RELEASE@amd64
Log cut:

2015-10-17T14:06:47.088+0000 I SHARDING [conn1] ChunkManager: time to load chunks for content.menu: 0ms sequenceNumber: 3 version: 1|0||55c68d69ba93c5f94b97e2d0 based on: (empty)
2015-10-17T14:06:52.363+0000 I SHARDING [conn7] ChunkManager: time to load chunks for auth.sessions: 0ms sequenceNumber: 4 version: 1|0||55c68248ba93c5f94b97e1b6 based on: (empty)
2015-10-17T14:06:52.363+0000 I SHARDING [conn7] ChunkManager: time to load chunks for auth.users: 0ms sequenceNumber: 5 version: 1|0||55c68274ba93c5f94b97e1bd based on: (empty)
2015-10-17T14:07:02.684+0000 I SHARDING [conn13] ChunkManager: time to load chunks for thumbnails.fs.chunks: 0ms sequenceNumber: 6 version: 3|1||55c68e3cba93c5f94b97e2ec based on: (empty)
2015-10-17T14:07:02.685+0000 I SHARDING [conn13] ChunkManager: time to load chunks for thumbnails.fs.files: 1ms sequenceNumber: 7 version: 6|1||55c68e8aba93c5f94b97e300 based on: (empty)
2015-10-17T14:07:16.269+0000 I -        [NetworkInterfaceASIO] Invariant failure connected src/mongo/executor/async_stream_common.h 55
2015-10-17T14:07:16.269+0000 I -        [NetworkInterfaceASIO] 
 
***aborting after invariant() failure
 
 
2015-10-17T14:07:16.270+0000 F -        [NetworkInterfaceASIO] Got signal: 6 (Abort trap).
 
 0xbea3db 0xbe9cab 0x80234747a 0x802347062
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"7EA3DB"},{"b":"400000","o":"7E9CAB"},{"b":"802339000","o":"E47A"},{"b":"802339000","o":"E062"}],"processInfo":{ "mongodbVersion" : "3.2.0-rc0", "gitVersion" : "nogitversion", "compiledModules" : [], "uname" : { "sysname" : "FreeBSD", "release" : "10.1-RELEASE", "version" : "FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC", "machine" : "amd64" } }}
 mongos(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x2B) [0xbea3db]
 mongos(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x11B) [0xbe9cab]
 libthr.so.3(_swapcontext+0x15A) [0x80234747a]
 libthr.so.3(sigaction+0x342) [0x802347062]
-----  END BACKTRACE  -----



 Comments   
Comment by Ramon Fernandez Marina [ 26/Nov/15 ]

Thanks for the update ashevchuk, glad to hear that you can build MongoDB again.

Regards,
Ramón.

Comment by Andrew Shevchuk [ 26/Nov/15 ]

Installed clang 3.7 binary package.
Building by clang 3.7 is solved this issue.
Thanks.

Comment by Andrew Shevchuk [ 24/Nov/15 ]

Yes, I can. I'll try and make a report.

Comment by Adam Midvidy [ 24/Nov/15 ]

Can you try building with clang-3.7?

Comment by Andrew Shevchuk [ 24/Nov/15 ]

Used for build, clang version information:
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Target: x86_64-unknown-freebsd10.1
Thread model: posix

Comment by Adam Midvidy [ 23/Nov/15 ]

Andrew - what is the toolchain you used to build the server?

I originally was able to reproduce the issue when building with clang++ 3.4, the version that ships with FreeBSD 10.2. After looking at some confusing core dumps, a coworker recommended that I try to build with clang++3.7. I installed llvm 3.7 from ports (devel/llvm37) and rebuilt - and to my surprise I could not reproduce the crash. If you are using the default toolchain can you try rebuilding your mongo processes with clang++3.7?

Comment by Adam Midvidy [ 23/Nov/15 ]

Hi Andrew,

Thanks for the information. I have reproduced this issue on FreeBSD 10.2 and am working on a fix.

Comment by Andrew Shevchuk [ 19/Nov/15 ]

Added an attachment with the mongos debug log while crash.
An accident happens every time, while starting the mongos process.

Comment by Adam Midvidy [ 18/Nov/15 ]

The interesting part of the trace is the fact that we ran out of memory. Are you able to reproduce the crash or was this a one-time issue? Additionally if the crash does reproduce, could you try running at a higher log verbosity (e.g. log level 2).

Comment by Andrew Shevchuk [ 18/Nov/15 ]

My Sharded cluster configuration is: 3 Config Servers, 3 Shards(without Replica Set), 1 Mongos Router.
All services was running in a separate Virtual Hosts(7 Virtual Hosts total) with a same configurations: 16 vCPU, 32G RAM, SSD storage, operating in VMware ESX Servers in vSphere.
Sharding servers used this additional configuration options: storage.engine: wiredTiger, sharding.archiveMovedChunks: false
Data in a cluster: storage is about 200Mb, 10 collections per 4 databases, 150-200 documments total, single GridFS database is about 95% of the total weight.
Used very simple queries, like "find" with simple conditions, but without MapReduce, $where, etc.
If needed, I can log all queries.

Comment by Andrew Shevchuk [ 18/Nov/15 ]

I can't build 3.2.0-rc3 at now, because of https://jira.mongodb.org/browse/SERVER-21216

Comment by Adam Midvidy [ 18/Nov/15 ]

Can you try again with 3.2.0-rc3? And can you give any details about the workload you are executing?

Generated at Thu Feb 08 03:56:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.