Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17888

FreeBSD mongod/mongos terminate when Clients Disconnect

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0.1
    • Component/s: Stability
    • Labels:
      None
    • FreeBSD
    • Hide

      Start the config servers on three machines:

      # ./mongod --configsvr --dbpath /data/mongodb/configdb                    
      2015-04-04T22:39:02.452+0200 W -        [initandlisten] Detected unclean shutdown - /data/mongodb/configdb/mongod.lock is not empty.
      2015-04-04T22:39:02.473+0200 I JOURNAL  [initandlisten] journal dir=/data/mongodb/configdb/journal
      2015-04-04T22:39:02.477+0200 I JOURNAL  [initandlisten] recover begin
      2015-04-04T22:39:02.482+0200 I JOURNAL  [initandlisten] recover lsn: 0
      2015-04-04T22:39:02.482+0200 I JOURNAL  [initandlisten] recover /data/mongodb/configdb/journal/j._0
      2015-04-04T22:39:02.486+0200 I STORAGE  [initandlisten] recover create file /data/mongodb/configdb/local.ns 16MB
      2015-04-04T22:39:02.503+0200 I STORAGE  [initandlisten] recover create file /data/mongodb/configdb/local.0 16MB
      2015-04-04T22:39:02.529+0200 I JOURNAL  [initandlisten] recover cleaning up
      2015-04-04T22:39:02.529+0200 I JOURNAL  [initandlisten] removeJournalFiles
      2015-04-04T22:39:02.529+0200 I JOURNAL  [initandlisten] recover done
      2015-04-04T22:39:02.630+0200 I JOURNAL  [durability] Durability thread started
      2015-04-04T22:39:02.631+0200 I CONTROL  [initandlisten] MongoDB starting : pid=768 port=27019 dbpath=/data/mongodb/configdb master=1 64-bit host=mon00.lab
      2015-04-04T22:39:02.631+0200 I CONTROL  [initandlisten] 
      2015-04-04T22:39:02.631+0200 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 8514 processes, 117243 files. Number of processes should be at least 58621.5 : 0.5 times number of files.
      2015-04-04T22:39:02.632+0200 I JOURNAL  [journal writer] Journal writer thread started
      2015-04-04T22:39:02.632+0200 I CONTROL  [initandlisten] db version v3.0.1
      2015-04-04T22:39:02.632+0200 I CONTROL  [initandlisten] git version: nogitversion
      2015-04-04T22:39:02.632+0200 I CONTROL  [initandlisten] build info: FreeBSD boot.lab 10.0-STABLE FreeBSD 10.0-STABLE #1 r264453: Mon Apr 14 09:46:40 CEST 2014     root@boot.lab:/usr/obj/usr/src/sys/GENERIC amd64 BOOST_LIB_VERSION=1_49
      2015-04-04T22:39:02.632+0200 I CONTROL  [initandlisten] allocator: system
      2015-04-04T22:39:02.632+0200 I CONTROL  [initandlisten] options: { sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/mongodb/configdb" } }
      2015-04-04T22:39:02.712+0200 I NETWORK  [initandlisten] waiting for connections on port 27019
      

      Then start mongos on the same machines with:

      # ./mongos --configdb mon00:27019,mon01:27019,mon02:27019
      

      This crashes all previously started mongod instances with:

      2015-04-04T22:49:20.159+0200 I NETWORK  [initandlisten] connection accepted from 172.16.133.195:33803 #1 (1 connection now open)
      2015-04-04T22:49:20.163+0200 I NETWORK  [initandlisten] connection accepted from 172.16.133.195:64234 #2 (2 connections now open)
      2015-04-04T22:49:20.173+0200 I NETWORK  [initandlisten] connection accepted from 172.16.133.196:24222 #3 (3 connections now open)
      2015-04-04T22:49:20.176+0200 F -        [conn2] terminate() called. An exception is active; attempting to gather more information
      2015-04-04T22:49:20.178+0200 F -        [conn2] DBException::toString(): 9001 socket exception [CLOSED] server [172.16.133.195:64234] 
      Actual exception type: mongo::SocketException
      
       0xf39952 0xf3967f 0x8021e87f8 0x8021e8843 0x8021e8a52 0xf05a3f 0xf06bba
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B39952"},{"b":"400000","o":"B3967F"},{"b":"80218A000","o":"5E7F8"},{"b":"80218A000","o":"5E843"},{"b":"80218A000","o":"5EA52"},{"b":"400000","o":"B05A3F"},{"b":"400000","o":"B06BBA"}],"processInfo":{ "mongodbVersion" : "3.0.1", "gitVersion" : "nogitversion", "uname" : { "sysname" : "FreeBSD", "release" : "10.1-STABLE", "version" : "FreeBSD 10.1-STABLE #32 r279517: Mon Mar  2 09:52:52 CET 2015     root@boot.lab:/data/obj/usr/src/sys/BOOTCLNT", "machine" : "amd64" } }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xf39952]
       mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x26F) [0xf3967f]
       libstdc++.so.6(_ZSt17rethrow_exceptionNSt15__exception_ptr13exception_ptrE+0x78) [0x8021e87f8]
       libstdc++.so.6(_ZSt9terminatev+0x13) [0x8021e8843]
       libstdc++.so.6(__cxa_throw+0x72) [0x8021e8a52]
       mongod(_ZN5mongo6Socket15handleRecvErrorEii+0x33F) [0xf05a3f]
       mongod(_ZN5mongo6Socket5_recvEPci+0x3A) [0xf06bba]
      -----  END BACKTRACE  -----
      
      Show
      Start the config servers on three machines: # ./mongod --configsvr --dbpath /data/mongodb/configdb 2015-04-04T22:39:02.452+0200 W - [initandlisten] Detected unclean shutdown - /data/mongodb/configdb/mongod.lock is not empty. 2015-04-04T22:39:02.473+0200 I JOURNAL [initandlisten] journal dir=/data/mongodb/configdb/journal 2015-04-04T22:39:02.477+0200 I JOURNAL [initandlisten] recover begin 2015-04-04T22:39:02.482+0200 I JOURNAL [initandlisten] recover lsn: 0 2015-04-04T22:39:02.482+0200 I JOURNAL [initandlisten] recover /data/mongodb/configdb/journal/j._0 2015-04-04T22:39:02.486+0200 I STORAGE [initandlisten] recover create file /data/mongodb/configdb/local.ns 16MB 2015-04-04T22:39:02.503+0200 I STORAGE [initandlisten] recover create file /data/mongodb/configdb/local.0 16MB 2015-04-04T22:39:02.529+0200 I JOURNAL [initandlisten] recover cleaning up 2015-04-04T22:39:02.529+0200 I JOURNAL [initandlisten] removeJournalFiles 2015-04-04T22:39:02.529+0200 I JOURNAL [initandlisten] recover done 2015-04-04T22:39:02.630+0200 I JOURNAL [durability] Durability thread started 2015-04-04T22:39:02.631+0200 I CONTROL [initandlisten] MongoDB starting : pid=768 port=27019 dbpath=/data/mongodb/configdb master=1 64-bit host=mon00.lab 2015-04-04T22:39:02.631+0200 I CONTROL [initandlisten] 2015-04-04T22:39:02.631+0200 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 8514 processes, 117243 files. Number of processes should be at least 58621.5 : 0.5 times number of files. 2015-04-04T22:39:02.632+0200 I JOURNAL [journal writer] Journal writer thread started 2015-04-04T22:39:02.632+0200 I CONTROL [initandlisten] db version v3.0.1 2015-04-04T22:39:02.632+0200 I CONTROL [initandlisten] git version: nogitversion 2015-04-04T22:39:02.632+0200 I CONTROL [initandlisten] build info: FreeBSD boot.lab 10.0-STABLE FreeBSD 10.0-STABLE #1 r264453: Mon Apr 14 09:46:40 CEST 2014 root@boot.lab:/usr/obj/usr/src/sys/GENERIC amd64 BOOST_LIB_VERSION=1_49 2015-04-04T22:39:02.632+0200 I CONTROL [initandlisten] allocator: system 2015-04-04T22:39:02.632+0200 I CONTROL [initandlisten] options: { sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/mongodb/configdb" } } 2015-04-04T22:39:02.712+0200 I NETWORK [initandlisten] waiting for connections on port 27019 Then start mongos on the same machines with: # ./mongos --configdb mon00:27019,mon01:27019,mon02:27019 This crashes all previously started mongod instances with: 2015-04-04T22:49:20.159+0200 I NETWORK [initandlisten] connection accepted from 172.16.133.195:33803 #1 (1 connection now open) 2015-04-04T22:49:20.163+0200 I NETWORK [initandlisten] connection accepted from 172.16.133.195:64234 #2 (2 connections now open) 2015-04-04T22:49:20.173+0200 I NETWORK [initandlisten] connection accepted from 172.16.133.196:24222 #3 (3 connections now open) 2015-04-04T22:49:20.176+0200 F - [conn2] terminate() called. An exception is active; attempting to gather more information 2015-04-04T22:49:20.178+0200 F - [conn2] DBException::toString(): 9001 socket exception [CLOSED] server [172.16.133.195:64234] Actual exception type: mongo::SocketException 0xf39952 0xf3967f 0x8021e87f8 0x8021e8843 0x8021e8a52 0xf05a3f 0xf06bba ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"B39952"},{"b":"400000","o":"B3967F"},{"b":"80218A000","o":"5E7F8"},{"b":"80218A000","o":"5E843"},{"b":"80218A000","o":"5EA52"},{"b":"400000","o":"B05A3F"},{"b":"400000","o":"B06BBA"}],"processInfo":{ "mongodbVersion" : "3.0.1", "gitVersion" : "nogitversion", "uname" : { "sysname" : "FreeBSD", "release" : "10.1-STABLE", "version" : "FreeBSD 10.1-STABLE #32 r279517: Mon Mar 2 09:52:52 CET 2015 root@boot.lab:/data/obj/usr/src/sys/BOOTCLNT", "machine" : "amd64" } }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xf39952] mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x26F) [0xf3967f] libstdc++.so.6(_ZSt17rethrow_exceptionNSt15__exception_ptr13exception_ptrE+0x78) [0x8021e87f8] libstdc++.so.6(_ZSt9terminatev+0x13) [0x8021e8843] libstdc++.so.6(__cxa_throw+0x72) [0x8021e8a52] mongod(_ZN5mongo6Socket15handleRecvErrorEii+0x33F) [0xf05a3f] mongod(_ZN5mongo6Socket5_recvEPci+0x3A) [0xf06bba] ----- END BACKTRACE -----

      I'm trying to set up a sharded mongodb instance with 3.0.1 on FreeBSD 10/amd64, without success. mongod crashes when mongos try to connect to it.
      The same setup works with 2.6.7 just fine.

            Assignee:
            sam.kleinman Sam Kleinman (Inactive)
            Reporter:
            bra@fsn.hu Nagy
            Votes:
            2 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: