[SERVER-26815] ShardRemote::getConnString() is very expensive Created: 27/Oct/16  Updated: 29/Jan/18  Resolved: 06/Sep/17

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Misha Tyulenev Assignee: Misha Tyulenev
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Participants:

 Description   

ShardRemote::getConnString() involves locking a mutex at ReplicaSetMonitor::getServerAddress and a building a string from scratch into a StringBuffer in SetState::getConfirmedServerAddress()
this should be cached to provide better performance and less lock contention.



 Comments   
Comment by Misha Tyulenev [ 22/Nov/16 ]

Per discussion with kaloian.manassiev according to the stack the call to getConnString() happens only on new connection establishment when the connection pool is being populated.

Comment by Misha Tyulenev [ 22/Nov/16 ]

In my experiment it was a mongo shell connected to a db. every find command was getting into this.

Comment by Kaloian Manassiev [ 22/Nov/16 ]

This only happens on new connection establishment, right?

Comment by Misha Tyulenev [ 22/Nov/16 ]

Done an investigation if getConnString() is on any CRUD path

collection.insert()
collection.update()
collection.remove()
do not call get getConnString()

however
collection.find() does call getConnString() as a part of the following stack trace

#0  mongo::ShardRemote::getConnString (this=0x7f639d1deec0) at src/mongo/s/client/shard_remote.cpp:119
#1  0x00007f63a0c0ee2d in mongo::ShardingNetworkConnectionHook::makeRequest (this=0x7f639d39db60, remoteHost=...)
    at src/mongo/s/client/sharding_network_connection_hook.cpp:106
#2  0x00007f63a0b0fb0e in mongo::executor::callNoexcept<mongo::executor::NetworkConnectionHook, mongo::StatusWith<boost::optional<mongo::executor::RemoteCommandRequest> >, mongo::HostAndPort const&, mongo::HostAndPort const&> (obj=..., 
    method=&virtual mongo::executor::NetworkConnectionHook::makeRequest(mongo::HostAndPort const&)) at src/mongo/executor/network_interface_asio.h:510
#3  0x00007f63a0b0c0b6 in mongo::executor::NetworkInterfaceASIO::_runConnectionHook (this=0x7f639d3b2b20, op=0x7f639cd134a0)
    at src/mongo/executor/network_interface_asio_command.cpp:437
#4  0x00007f63a0b06852 in mongo::executor::NetworkInterfaceASIO::_authenticate (this=0x7f639d3b2b20, op=0x7f639cd134a0)
    at src/mongo/executor/network_interface_asio_auth.cpp:179
#5  0x00007f63a0b0592b in mongo::executor::NetworkInterfaceASIO::<lambda()>::operator()(void) const (__closure=0x7f639f3a4cb0)
    at src/mongo/executor/network_interface_asio_auth.cpp:162
#6  0x00007f63a0b06d3f in mongo::executor::NetworkInterfaceASIO::_validateAndRun<const mongo::executor::NetworkInterfaceASIO::_runIsMaster(mongo::executor::NetworkInterfaceASIO::AsyncOp*)::<lambda()> >(mongo::executor::NetworkInterfaceASIO::AsyncOp *, std::error_code, <unknown type in /home/misha/work/mongo_2/mongos, CU 0x421fe21, DIE 0x425cdb1>) (this=0x7f639d3b2b20, op=0x7f639cd134a0, ec=..., 
    handler=<unknown type in /home/misha/work/mongo_2/mongos, CU 0x421fe21, DIE 0x425cdb1>) at src/mongo/executor/network_interface_asio.h:435
#7  0x00007f63a0b05c75 in mongo::executor::NetworkInterfaceASIO::<lambda(std::error_code, size_t)>::operator()(std::error_code, size_t) const (
    __closure=0x7f639f3a4ca0, ec=..., bytes=189) at src/mongo/executor/network_interface_asio_auth.cpp:167
#8  0x00007f63a0b0758a in std::_Function_handler<void(std::error_code, long unsigned int), mongo::executor::NetworkInterfaceASIO::_runIsMaster(mongo::executor::NetworkInterfaceASIO::AsyncOp*)::<lambda(std::error_code, size_t)> >::_M_invoke(const std::_Any_data &, <unknown type in /home/misha/work/mongo_2/mongos, CU 0x421fe21, DIE 0x425e334>, <unknown type in /home/misha/work/mongo_2/mongos, CU 0x421fe21, DIE 0x425e339>) (__functor=..., 
    __args#0=<unknown type in /home/misha/work/mongo_2/mongos, CU 0x421fe21, DIE 0x425e334>, 
    __args#1=<unknown type in /home/misha/work/mongo_2/mongos, CU 0x421fe21, DIE 0x425e339>) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:1871
#9  0x00007f63a0abeca3 in std::function<void (std::error_code, unsigned long)>::operator()(std::error_code, unsigned long) const (this=0x7f63893780b0, 
    __args#0=..., __args#1=189) at /opt/mongodbtoolchain/v2/include/c++/5.3.0/functional:2271
#10 0x00007f63a0b0af88 in mongo::executor::NetworkInterfaceASIO::<lambda(std::error_code, size_t)>::operator()(std::error_code, size_t) const (
    __closure=0x7f63893780a0, ec=..., bytes=189) at src/mongo/executor/network_interface_asio_command.cpp:390
#11 0x00007f63a0b0e822 in std::_Function_handler<void(std::error_code, long unsigned int), mongo::executor::NetworkInterfaceASIO::_asyncRunCommand(mongo::executor::NetworkInterfaceASIO::AsyncOp*, mongo::executor::NetworkInterfaceASIO::NetworkOpHandler)::<lambda(std::error_code, size_t)> >::_M_invoke(const std::_Any_data &, <unknown type in /home/misha/work/mongo_2/mongos, CU 0x4260d54, DIE 0x42aa0ca>, <unknown type in /home/misha/work/mongo_2/mongos, CU 0x4260d54, DIE 0x42aa0cf>) (__functor=..., __args#0=<unknown type in /home/misha/work/mongo_2/mongos, CU 0x4260d54, DIE 0x42aa0ca>, 

Comment by Kaloian Manassiev [ 28/Oct/16 ]

Chunk::split is a fairly rare operation - to the extent that caching is not necessary. Can you please update the ticket description with the operations which are impacted?

Comment by Misha Tyulenev [ 28/Oct/16 ]

kaloian.manassiev Its being called a lot - e.g its a part of Chhunk::split. Not sure if its a part of any CRUD. Its new, in 3.2 Shard just returned the cached CS.

Comment by Kaloian Manassiev [ 28/Oct/16 ]

misha.tyulenev - what is the external impact of this? Is it being called on critical code paths and does impact CRUD operations or just maintenance calls and new connection establishment? Also - is it new in 3.4?

Generated at Thu Feb 08 04:13:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.