[SERVER-17888] FreeBSD mongod/mongos terminate when Clients Disconnect Created: 04/Apr/15  Updated: 09/Oct/15  Resolved: 09/Oct/15

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 3.0.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Nagy Assignee: Sam Kleinman (Inactive)
Resolution: Duplicate Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
duplicates SERVER-17747 FreeBSD 11.0-CURRENT build issue Closed
Operating System: FreeBSD
Steps To Reproduce:

Start the config servers on three machines:

# ./mongod --configsvr --dbpath /data/mongodb/configdb                    
2015-04-04T22:39:02.452+0200 W -        [initandlisten] Detected unclean shutdown - /data/mongodb/configdb/mongod.lock is not empty.
2015-04-04T22:39:02.473+0200 I JOURNAL  [initandlisten] journal dir=/data/mongodb/configdb/journal
2015-04-04T22:39:02.477+0200 I JOURNAL  [initandlisten] recover begin
2015-04-04T22:39:02.482+0200 I JOURNAL  [initandlisten] recover lsn: 0
2015-04-04T22:39:02.482+0200 I JOURNAL  [initandlisten] recover /data/mongodb/configdb/journal/j._0
2015-04-04T22:39:02.486+0200 I STORAGE  [initandlisten] recover create file /data/mongodb/configdb/local.ns 16MB
2015-04-04T22:39:02.503+0200 I STORAGE  [initandlisten] recover create file /data/mongodb/configdb/local.0 16MB
2015-04-04T22:39:02.529+0200 I JOURNAL  [initandlisten] recover cleaning up
2015-04-04T22:39:02.529+0200 I JOURNAL  [initandlisten] removeJournalFiles
2015-04-04T22:39:02.529+0200 I JOURNAL  [initandlisten] recover done
2015-04-04T22:39:02.630+0200 I JOURNAL  [durability] Durability thread started
2015-04-04T22:39:02.631+0200 I CONTROL  [initandlisten] MongoDB starting : pid=768 port=27019 dbpath=/data/mongodb/configdb master=1 64-bit host=mon00.lab
2015-04-04T22:39:02.631+0200 I CONTROL  [initandlisten] 
2015-04-04T22:39:02.631+0200 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 8514 processes, 117243 files. Number of processes should be at least 58621.5 : 0.5 times number of files.
2015-04-04T22:39:02.632+0200 I JOURNAL  [journal writer] Journal writer thread started
2015-04-04T22:39:02.632+0200 I CONTROL  [initandlisten] db version v3.0.1
2015-04-04T22:39:02.632+0200 I CONTROL  [initandlisten] git version: nogitversion
2015-04-04T22:39:02.632+0200 I CONTROL  [initandlisten] build info: FreeBSD boot.lab 10.0-STABLE FreeBSD 10.0-STABLE #1 r264453: Mon Apr 14 09:46:40 CEST 2014     root@boot.lab:/usr/obj/usr/src/sys/GENERIC amd64 BOOST_LIB_VERSION=1_49
2015-04-04T22:39:02.632+0200 I CONTROL  [initandlisten] allocator: system
2015-04-04T22:39:02.632+0200 I CONTROL  [initandlisten] options: { sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/mongodb/configdb" } }
2015-04-04T22:39:02.712+0200 I NETWORK  [initandlisten] waiting for connections on port 27019

Then start mongos on the same machines with:

# ./mongos --configdb mon00:27019,mon01:27019,mon02:27019

This crashes all previously started mongod instances with:

2015-04-04T22:49:20.159+0200 I NETWORK  [initandlisten] connection accepted from 172.16.133.195:33803 #1 (1 connection now open)
2015-04-04T22:49:20.163+0200 I NETWORK  [initandlisten] connection accepted from 172.16.133.195:64234 #2 (2 connections now open)
2015-04-04T22:49:20.173+0200 I NETWORK  [initandlisten] connection accepted from 172.16.133.196:24222 #3 (3 connections now open)
2015-04-04T22:49:20.176+0200 F -        [conn2] terminate() called. An exception is active; attempting to gather more information
2015-04-04T22:49:20.178+0200 F -        [conn2] DBException::toString(): 9001 socket exception [CLOSED] server [172.16.133.195:64234] 
Actual exception type: mongo::SocketException
 
 0xf39952 0xf3967f 0x8021e87f8 0x8021e8843 0x8021e8a52 0xf05a3f 0xf06bba
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B39952"},{"b":"400000","o":"B3967F"},{"b":"80218A000","o":"5E7F8"},{"b":"80218A000","o":"5E843"},{"b":"80218A000","o":"5EA52"},{"b":"400000","o":"B05A3F"},{"b":"400000","o":"B06BBA"}],"processInfo":{ "mongodbVersion" : "3.0.1", "gitVersion" : "nogitversion", "uname" : { "sysname" : "FreeBSD", "release" : "10.1-STABLE", "version" : "FreeBSD 10.1-STABLE #32 r279517: Mon Mar  2 09:52:52 CET 2015     root@boot.lab:/data/obj/usr/src/sys/BOOTCLNT", "machine" : "amd64" } }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xf39952]
 mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x26F) [0xf3967f]
 libstdc++.so.6(_ZSt17rethrow_exceptionNSt15__exception_ptr13exception_ptrE+0x78) [0x8021e87f8]
 libstdc++.so.6(_ZSt9terminatev+0x13) [0x8021e8843]
 libstdc++.so.6(__cxa_throw+0x72) [0x8021e8a52]
 mongod(_ZN5mongo6Socket15handleRecvErrorEii+0x33F) [0xf05a3f]
 mongod(_ZN5mongo6Socket5_recvEPci+0x3A) [0xf06bba]
-----  END BACKTRACE  -----

Participants:

 Description   

I'm trying to set up a sharded mongodb instance with 3.0.1 on FreeBSD 10/amd64, without success. mongod crashes when mongos try to connect to it.
The same setup works with 2.6.7 just fine.



 Comments   
Comment by Andrew Aldridge [ 08/Oct/15 ]

After backporting 014d0c666ec15e819ad2bedf32a850ae29ede26e to the v3.0 branch and building on FreeBSD 10.2, I haven't yet been able to reproduce this crash.

Marking as a duplicate of SERVER-17747. Will check with jonathan.reams/acm if we can schedule for backport to 3.0.8.

Comment by Daniel Pasette (Inactive) [ 29/Sep/15 ]

The "catch (SocketException& e)" in handleIncomingMsg() ought to be catching that exception. Figuring out why that isn't happening is the key to debugging this. Maybe FreeBSD has a bug in their C++ library?

Comment by Lorenzo P. [ 27/Sep/15 ]

Hi, it would be encouraging to see some progress on this issue. This is most probably the number one reason why we're still waiting for a 3.x port in the ports tree.
Best Regards

Comment by Kaspars Bankovskis [ 27/Aug/15 ]

to make it more clear - issue is not limited only to the specific use case described above. I reproduced it in the simplest setup, by starting mongod (mongod --dbpath /path/to/empty/dir) and connecting with mongo shell from another terminal.

Comment by Kaspars Bankovskis [ 27/Aug/15 ]

Any progress with this issue? The problem is still present with mongodb v3.0.6 when compiled on FreeBSD 10.2 amd64.

Comment by Sam Kleinman (Inactive) [ 13/Apr/15 ]

Thanks for this data. I've retitled the ticket slightly, and we'll continue to investigate.

Comment by Nagy [ 06/Apr/15 ]

I've tried mongodb 3 on at least five machines until now, memory ranging from 4G to 32, nothing else runs.
This is all I get with the default settings:

2015-04-06T19:38:33.422+0200 I JOURNAL  [initandlisten] journal dir=/usr/home/mongodb/db/journal
2015-04-06T19:38:33.424+0200 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
2015-04-06T19:38:33.495+0200 I JOURNAL  [durability] Durability thread started
2015-04-06T19:38:33.495+0200 I CONTROL  [initandlisten] MongoDB starting : pid=80232 port=27017 dbpath=/usr/home/mongodb/db 64-bit host=fbsd
2015-04-06T19:38:33.496+0200 I CONTROL  [initandlisten] 
2015-04-06T19:38:33.496+0200 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 12200 processes, 235161 files. Number of processes should be at least 117580 : 0.5 times number of files.
2015-04-06T19:38:33.496+0200 I CONTROL  [initandlisten] db version v3.0.1
2015-04-06T19:38:33.496+0200 I CONTROL  [initandlisten] git version: nogitversion
2015-04-06T19:38:33.496+0200 I JOURNAL  [journal writer] Journal writer thread started
2015-04-06T19:38:33.496+0200 I CONTROL  [initandlisten] build info: FreeBSD boot.lab 10.0-STABLE FreeBSD 10.0-STABLE #1 r264453: Mon Apr 14 09:46:40 CEST 2014     root@boot.lab:/usr/obj/usr/src/sys/GENERIC amd64 BOOST_LIB_VERSION=1_49
2015-04-06T19:38:33.497+0200 I CONTROL  [initandlisten] allocator: system
2015-04-06T19:38:33.497+0200 I CONTROL  [initandlisten] options: { storage: { dbPath: "/usr/home/mongodb/db" } }
2015-04-06T19:38:33.498+0200 I INDEX    [initandlisten] allocating new ns file /usr/home/mongodb/db/local.ns, filling with zeroes...
2015-04-06T19:38:33.511+0200 I STORAGE  [FileAllocator] allocating new datafile /usr/home/mongodb/db/local.0, filling with zeroes...
2015-04-06T19:38:33.511+0200 I STORAGE  [FileAllocator] creating directory /usr/home/mongodb/db/_tmp
2015-04-06T19:38:33.511+0200 I STORAGE  [FileAllocator] done allocating datafile /usr/home/mongodb/db/local.0, size: 64MB,  took 0 secs
2015-04-06T19:38:33.532+0200 I NETWORK  [initandlisten] waiting for connections on port 27017
2015-04-06T19:38:47.299+0200 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:62396 #1 (1 connection now open)
2015-04-06T19:38:48.786+0200 F -        [conn1] terminate() called. An exception is active; attempting to gather more information
2015-04-06T19:38:48.789+0200 F -        [conn1] DBException::toString(): 9001 socket exception [CLOSED] server [127.0.0.1:62396] 
Actual exception type: mongo::SocketException
 
 0xf39952 0xf3967f 0x8021e63b8 0x8021e6403 0x8021e6612 0xf05a3f 0xf06bba
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B39952"},{"b":"400000","o":"B3967F"},{"b":"802188000","o":"5E3B8"},{"b":"802188000","o":"5E403"},{"b":"802188000","o":"5E612"},{"b":"400000","o":"B05A3F"},{"b":"400000","o":"B06BBA"}],"processInfo":{ "mongodbVersion" : "3.0.1", "gitVersion" : "nogitversion", "uname" : { "sysname" : "FreeBSD", "release" : "10.1-RELEASE", "version" : "FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC", "machine" : "amd64" } }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xf39952]
 mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x26F) [0xf3967f]
 libstdc++.so.6(_ZSt17rethrow_exceptionNSt15__exception_ptr13exception_ptrE+0x78) [0x8021e63b8]
 libstdc++.so.6(_ZSt9terminatev+0x13) [0x8021e6403]
 libstdc++.so.6(__cxa_throw+0x72) [0x8021e6612]
 mongod(_ZN5mongo6Socket15handleRecvErrorEii+0x33F) [0xf05a3f]
 mongod(_ZN5mongo6Socket5_recvEPci+0x3A) [0xf06bba]
-----  END BACKTRACE  -----

First, I've tried to set up a sharded replica set setup, I don't know whether those processes involve disconnect or not, And yes, connecting and disconnecting with the mongo shell is enough to get the crash. mongod crashes when I disconnect.

If you send me an ssh public key privately, I can give you an account where you can try this if you don't have a FreeBSD box at hand.

Comment by Sam Kleinman (Inactive) [ 06/Apr/15 ]

To help us begin to debug this issue, would you be able to provide some additional information:

  • Could you describe the system where you're getting this issue. The backtrace indicates that you might be out of memory, (reportOutOfMemoryErrorAndExit}), but understanding the system's resource, configuration and any other process that may be running on the system.
  • Could you provide logs from the mongod instance before and during the crash?
  • Can you reproduce the issue without inserting data? Is connecting and disconnecting, enough to trigger the issue?
  • Can you reproduce this using the mongo shell? If not what version of pymongo are you using to reproduce that issue?

Sorry about this, and I hope we can get this sorted out quickly.

Cheers,
sam

Comment by Nagy [ 05/Apr/15 ]

I've started a single mongod instance and connected to it with a pymongo program. I can insert documents, but when the client disconnects, mongod dies with the same error, so it has nothing to do with mongos.

Generated at Thu Feb 08 03:45:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.