Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-4576

mongos stack traces after bind failure

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • None
    • None
    • None
    • ALL

    Description

      This happened while running sharding/auth.js on my machine with --dd and --durableDefaultOff options

       
      shell: started program /Users/aaron/src/mongo/mongos --port 31000 --configdb Aaron-Staples-MacBook-Pro.local:40000 --keyFile jstests/libs/key1 --chunkSize 1
       m31000| Wed Dec 28 14:21:13 Warning: soft rlimits too low. Number of files is 256, should be at least 1000
       m31000| Wed Dec 28 14:21:13 /Users/aaron/src/mongo/mongos db version v2.1.0-pre-, pdfile version 4.5 starting (--help for usage)
       m31000| Wed Dec 28 14:21:13 git version: ff70c7c017840ec54897085e7d0bc5d4f8f5acb7
       m31000| Wed Dec 28 14:21:13 build info: Darwin Aaron-Staples-MacBook-Pro.local 11.0.0 Darwin Kernel Version 11.0.0: Sat Jun 18 12:56:35 PDT 2011; root:xnu-1699.22.73~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_47
       m31000| Wed Dec 28 14:21:13 [websvr] ERROR: listen(): bind() failed errno:48 Address already in use for socket: 0.0.0.0:32000
       m31000| Wed Dec 28 14:21:13 [websvr] ERROR:   addr already in use
       m31000| Wed Dec 28 14:21:13 [Balancer] about to contact config servers and shards
       m31000| Wed Dec 28 14:21:13 [Balancer] config servers and shards contacted successfully
       m31000| Wed Dec 28 14:21:13 [Balancer] balancer id: Aaron-Staples-MacBook-Pro.local:31000 started at Dec 28 14:21:13
       m31000| Wed Dec 28 14:21:13 [Balancer] created new distributed lock for balancer on Aaron-Staples-MacBook-Pro.local:40000 ( lock timeout : 900000, ping interval : 30000, process : 0 )
       m31000| Wed Dec 28 14:21:13 [Balancer] creating WriteBackListener for: Aaron-Staples-MacBook-Pro.local:40000 serverID: 4efb9659f52096cef2142469
       m31000| Wed Dec 28 14:21:13 [mongosMain] ERROR: listen(): bind() failed errno:48 Address already in use for socket: 0.0.0.0:31000
       m31000| Wed Dec 28 14:21:13 [mongosMain] ERROR:   addr already in use
       m31000| Wed Dec 28 14:21:13 [mongosMain] dbexit:  rc:0 
       m31000| _DEBUG warning trying to lock a mongo::mutex during static shutdown
       m31000| 0x105b5a005 0x105b746df 0x105b52b28 0x105b9af14 0x105b962b7 0x105bc6c5a 0x105bc6c7b 0x105bc9af3 0x105ba3523 0x105badb0d 0x105badbbe 0x105badc0c 0x105badcca 0x105badd09 0x105badd8c 0x105ba5c26 0x105ba61fb 0x7fff8b35b7c8 0x7fff8b35b652 0x105fce45c 
       m31000| Received signal 11
       m31000| Backtrace: 0x105b873bd 0x105b874ca 0x7fff8b3bccfa 0x7ff29840f7b8 0x105f48b42 0x105ba346b 0x105ba40cd 0x105ba6e86 0x105c35c2d 0x105c314dc 0x105c31a2f 0x105ef62b3 0x105fe820f 0x105b4c3cd 0x105b50b83 0x105b50c82 0x105b50d54 0x105b50d7f 0x106bde460 0x7fff8b3688bf 
       m40000| Wed Dec 28 14:21:13 [initandlisten] connection accepted from 10.1.10.180:52495 #10 (10 connections now open)
       m40000| Wed Dec 28 14:21:13 [conn10]  authenticate: { authenticate: 1, nonce: "262cc0c05fd82b93", user: "__system", key: "38dbee07e6ce8ed25ab02a171e0608ec" }
       m40000| Wed Dec 28 14:21:13 [initandlisten] connection accepted from 10.1.10.180:52496 #11 (11 connections now open)
       m40000| Wed Dec 28 14:21:13 [conn11]  authenticate: { authenticate: 1, nonce: "2c00d67641d56c51", user: "__system", key: "7460408a1c2d1cb0c6561c15ae088e15" }
       m40000| Wed Dec 28 14:21:13 [initandlisten] connection accepted from 10.1.10.180:52497 #12 (12 connections now open)
       m40000| Wed Dec 28 14:21:13 [conn12]  authenticate: { authenticate: 1, nonce: "50b752ae077a597f", user: "__system", key: "8758e26fb74e1733d245735c34bd15ea" }
       m40000| Wed Dec 28 14:21:13 [conn10] end connection 10.1.10.180:52495 (11 connections now open)
       m40000| Wed Dec 28 14:21:13 [conn10] thread conn stack usage was 15256 bytes
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       m31000| WARNING: mongod wrote null bytes to output
       
       m31000| 0   mongos                              0x0000000105b873bd _ZN5mongoL18formattedBacktraceEi + 451   mongos                              0x0000000105b874ca _ZN5mongo17printStackAndExitEi + 902   libsystem_c.dylib                   0x00007fff8b3bccfa _sigtramp + 263   ???                                 0x00007ff29840f7b8 0x0 + 1406799132077364   mongos                              0x0000000105f48b42 _ZN5mongo22ShardingConnectionHook8onCreateEPNS_12DBClientBaseE + 11865   mongos                              0x0000000105ba346b _ZN5mongo16DBConnectionPool8onCreateEPNS_12DBClientBaseE + 1356   mongos                              0x0000000105ba40cd _ZN5mongo16DBConnectionPool13_finishCreateERKSsdPNS_12DBClientBaseE + 3957   mongos                              0x0000000105ba6e86 _ZN5mongo16DBConnectionPool3getERKSsd + 19528   mongos                              0x0000000105c35c2d _ZN5mongo17ClientConnections3getERKSsS2_ + 3799   mongos                              0x0000000105c314dc _ZN5mongo15ShardConnection5_initEv + 15210  mongos                              0x0000000105c31a2f _ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSsN5boost10shared_ptrIKNS_12ChunkManagerEEE + 17711  mongos                              0x0000000105ef62b3 _ZNK5mongo4Grid13shouldBalanceEv + 187 0   mongos                              0x0000000105b5a005 _ZN5mongo15printStackTraceERSo + 37
       m31000|  1   mongos                              0x0000000105b746df _ZN5mongo5mutex11scoped_lock15PostStaticCheckC1Ev + 87
       m31000|  2   mongos                              0x0000000105b52b28 _ZN5mongo5mutex11scoped_lockC1ERS0_ + 40
       m31000|  3   mongos                              0x0000000105b9af14 _ZN5mongo5Ports5eraseEPNS_13MessagingPortE + 48
       m31000|  4   mongos                              0x0000000105b962b7 _ZN5mongo13MessagingPortD0Ev + 143
       m31000|  5   mongos                              0x0000000105bc6c5a _ZN5boost14checked_deleteIN5mongo13MessagingPortEEEvPT_ + 42
       m31000|  6   mongos                              0x0000000105bc6c7b _ZN5boost10scoped_ptrIN5mongo13MessagingPortEED1Ev + 27
       m31000|  7   mongos                              0x0000000105bc9af3 _ZN5mongo18DBClientConnectionD0Ev + 519
       m31000|  8   mongos                              0x0000000105ba3523 _ZN5mongo11PoolForHostD1Ev + 85
       m31000|  9   mongos                              0x0000000105badb0d _ZNSt4pairIKN5mongo16DBConnectionPool7PoolKeyENS0_11PoolForHostEED1Ev + 37
       m31000|  10  mongos                              0x0000000105badbbe _ZN9__gnu_cxx13new_allocatorISt4pairIKN5mongo16DBConnectionPool7PoolKeyENS2_11PoolForHostEEE7destroyEPS7_ + 28
       m31000|  11  mongos                              0x0000000105badc0c _ZNSt8_Rb_treeIN5mongo16DBConnectionPool7PoolKeyESt4pairIKS2_NS0_11PoolForHostEESt10_Select1stIS6_ENS1_14poolKeyCompareESaIS6_EE15_M_destroy_nodeEPSt13_Rb_tree_nodeIS6_E + 72
       m31000|  12  mongos                              0x0000000105badcca _ZNSt8_Rb_treeIN5mongo16DBConnectionPool7PoolKeyESt4pairIKS2_NS0_11PoolForHostEESt10_Select1stIS6_ENS1_14poolKeyCompareESaIS6_EE8_M_eraseEPSt13_Rb_tree_nodeIS6_E + 80
       m31000|  13  mongos                              0x0000000105badd09 _ZNSt8_Rb_treeIN5mongo16DBConnectionPool7PoolKeyESt4pairIKS2_NS0_11PoolForHostEESt10_Select1stIS6_ENS1_14poolKeyCompareESaIS6_EED1Ev + 39
       m31000|  14  mongos                              0x0000000105badd8c _ZNSt3mapIN5mongo16DBConnectionPool7PoolKeyENS0_11PoolForHostENS1_14poolKeyCompareESaISt4pairIKS2_S3_EEED1Ev + 24
       m31000|  15  mongos                              0x0000000105ba5c26 _ZN5mongo16DBConnectionPoolD1Ev + 64
       m31000|  16  mongos                              0x0000000105ba61fb __tcf_6 + 27
       m31000|  17  libsystem_c.dylib                   0x00007fff8b35b7c8 __cxa_finalize + 274
       m31000|  18  libsystem_c.dylib                   0x00007fff8b35b652 exit + 18
       m31000|  19  mongos                              0x0000000105fce45c _ZN5mongo6dbexitENS_8ExitCodeEPKcb + 316

      Seems like there is an issue relating to locking a mutex during destruction of a connection pool, and also potentially another issue in the sharding connection hook's onCreate(). And also it looks like there is a problem relating to printing two stack traces at the same time. But I think maybe stack printing should be pretty light weight don't know if anything can/should be done about that.

      Attachments

        Activity

          People

            spencer@mongodb.com Spencer Brody (Inactive)
            aaron Aaron Staple
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: