-
Type:
New Feature
-
Resolution: Won't Fix
-
Priority:
Major - P3
-
None
-
Affects Version/s: 3.0
-
Component/s: None
-
Labels:None
-
Environment:pymongo==3.0.1-3-70185ff
Summary: In the reporter's network, an "ismaster" call to a remote mongos is actually lower latency than to a mongos running on localhost, but the local mongos has higher throughput. PyMongo does not measure throughput, only latency, and it prefers the lowest-latency mongos. This is a feature request to work around this issue by especially preferring "localhost" and its synonyms, regardless of latency measurements.
We've updated the Server Selection Spec to mandate all drivers prefer localhost if the Topology Type is sharded.
Original report
It looks like pymongo 3.0.1.dev0 is consistently slower than 2.6.3 for find:
3.0.1.dev0
In [42]: %timeit -n10 list(coll.find({'symbol': 'FUT_FTL', 'parent': ObjectId('5535d898cdc41bd41e2a60be')}))
10 loops, best of 3: 293 ms per loop
In [43]: %timeit -n10 list(coll.find({'symbol': 'FUT_FTL', 'parent': ObjectId('5535d898cdc41bd41e2a60be')}))
10 loops, best of 3: 241 ms per loop
In [44]: %timeit -n10 list(coll.find({'symbol': 'FUT_FTL', 'parent': ObjectId('5535d898cdc41bd41e2a60be')}))
10 loops, best of 3: 291 ms per loop
In [45]: %timeit -n10 list(coll.find({'symbol': 'FUT_FTL', 'parent': ObjectId('5535d898cdc41bd41e2a60be')}))
10 loops, best of 3: 309 ms per loop
Avg: 283.5 ms
2.6.3
In [13]: coll = m['oneminute.STITCHED']._collection 2015-04-21 08:59:07,673 INFO 14654 Connecting to mongo: research (cn15-ib:27119,cn16-ib:27119,localhost:27119) In [14]: %timeit -n10 list(coll.find({'symbol': 'FUT_FTL', 'parent': ObjectId('5535d898cdc41bd41e2a60be')})) 10 loops, best of 3: 196 ms per loop In [15]: %timeit -n10 list(coll.find({'symbol': 'FUT_FTL', 'parent': ObjectId('5535d898cdc41bd41e2a60be')})) 10 loops, best of 3: 214 ms per loop In [16]: %timeit -n10 list(coll.find({'symbol': 'FUT_FTL', 'parent': ObjectId('5535d898cdc41bd41e2a60be')})) 10 loops, best of 3: 184 ms per loop In [17]: %timeit -n10 list(coll.find({'symbol': 'FUT_FTL', 'parent': ObjectId('5535d898cdc41bd41e2a60be')})) 10 loops, best of 3: 216 ms per loop
Avg: 202.5
With prun, I see that pymongo 3.0 is performing 346 socket recv's vs 81 for 2.6.3. Has something changed with the socket recv buffer size, or similar? I haven't been able to find anything in the docs.
3.0.1.dev0
%prun x= list(coll.find({'symbol': 'FUT_FTL', 'parent': ObjectId('5535d898cdc41bd41e2a60be')}))
7763 function calls (7747 primitive calls) in 0.689 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
346 0.633 0.002 0.633 0.002 {method 'recv' of '_socket.socket' objects}
16 0.024 0.001 0.657 0.041 network.py:73(_receive_data_on_socket)
8 0.009 0.001 0.020 0.003 {bson._cbson.decode_all}
696 0.006 0.000 0.006 0.000 {built-in method __new__ of type object at 0x2b63d125b880}
8 0.005 0.001 0.025 0.003 helpers.py:87(_unpack_response)
696 0.003 0.000 0.009 0.000 binary.py:141(__new__)
697 0.001 0.000 0.002 0.000 objectid.py:68(__init__)
233 0.001 0.000 0.689 0.003 cursor.py:972(__next__)
232 0.001 0.000 0.001 0.000 database.py:339(_fix_outgoing)
2099 0.001 0.000 0.001 0.000 {isinstance}
1321 0.001 0.000 0.001 0.000 {len}
1 0.000 0.000 0.689 0.689 <string>:1(<module>)
8 0.000 0.000 0.660 0.082 server.py:66(send_message_with_response)
8 0.000 0.000 0.687 0.086 cursor.py:792(__send_message)
8 0.000 0.000 0.000 0.000 {method 'sendall' of '_socket.socket' objects}
8 0.000 0.000 0.661 0.083 mongo_client.py:692(_send_message_with_response)
8 0.000 0.000 0.000 0.000 topology.py:61(select_servers)
241 0.000 0.000 0.000 0.000 collection.py:249(database)
16/8 0.000 0.000 0.001 0.000 contextlib.py:21(__exit__)
8 0.000 0.000 0.000 0.000 pool.py:521(_get_socket_no_auth)
8 0.000 0.000 0.000 0.000 pool.py:556(return_socket)
9 0.000 0.000 0.687 0.076 cursor.py:884(_refresh)
1 0.000 0.000 0.000 0.000 {select.select}
8 0.000 0.000 0.657 0.082 network.py:58(receive_message)
16 0.000 0.000 0.001 0.000 pool.py:484(get_socket)
16 0.000 0.000 0.001 0.000 server.py:100(get_socket)
8 0.000 0.000 0.000 0.000 topology.py:306(_apply_selector)
32 0.000 0.000 0.000 0.000 periodic_executor.py:46(open)
8 0.000 0.000 0.000 0.000 pool.py:257(check_auth)
32 0.000 0.000 0.000 0.000 threading.py:696(isAlive)
8 0.000 0.000 0.000 0.000 topology.py:265(_ensure_opened)
7 0.000 0.000 0.000 0.000 {pymongo._cmessage._get_more_message}
2.6.3
9032 function calls in 0.206 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
83 0.145 0.002 0.145 0.002 {method 'recv' of '_socket.socket' objects}
16 0.017 0.001 0.162 0.010 mongo_client.py:983(__receive_data_on_socket)
8 0.011 0.001 0.036 0.004 helpers.py:74(_unpack_response)
8 0.010 0.001 0.025 0.003 {bson._cbson.decode_all}
696 0.007 0.000 0.007 0.000 {built-in method __new__ of type object at 0x2ab756479880}
696 0.003 0.000 0.011 0.000 binary.py:130(__new__)
697 0.002 0.000 0.003 0.000 objectid.py:169(__validate)
1 0.002 0.002 0.206 0.206 <string>:1(<module>)
3492 0.001 0.000 0.001 0.000 {isinstance}
9 0.001 0.000 0.202 0.022 cursor.py:826(_refresh)
233 0.001 0.000 0.204 0.001 cursor.py:900(next)
697 0.001 0.000 0.004 0.000 objectid.py:68(__init__)
232 0.001 0.000 0.001 0.000 database.py:259(_fix_outgoing)
1058 0.000 0.000 0.000 0.000 {len}
8 0.000 0.000 0.201 0.025 cursor.py:761(__send_message)
8 0.000 0.000 0.163 0.020 mongo_client.py:998(__receive_message_on_socket)
8 0.000 0.000 0.000 0.000 {method 'sendall' of '_socket.socket' objects}
8 0.000 0.000 0.001 0.000 pool.py:286(get_socket)
40 0.000 0.000 0.000 0.000 {method 'acquire' of 'thread.lock' objects}
8 0.000 0.000 0.001 0.000 pool.py:393(maybe_return_socket)
8 0.000 0.000 0.165 0.021 mongo_client.py:1027(_send_message_with_response)
16 0.000 0.000 0.000 0.000 thread_util.py:90(_make_vigil)
24 0.000 0.000 0.000 0.000 {hasattr}
8 0.000 0.000 0.163 0.020 mongo_client.py:1014(__send_and_receive)
245 0.000 0.000 0.000 0.000 collection.py:174(database)
7 0.000 0.000 0.000 0.000 {pymongo._cmessage._get_more_message}
8 0.000 0.000 0.001 0.000 mongo_client.py:752(__socket)
8 0.000 0.000 0.000 0.000 mongo_client.py:454(__check_auth)
16 0.000 0.000 0.000 0.000 pool.py:517(_get_request_state)
8 0.000 0.000 0.000 0.000 pool.py:414(_return_socket)
1 0.000 0.000 0.000 0.000 cursor.py:66(__init__)
56 0.000 0.000 0.000 0.000 {_struct.unpack}
- is depended on by
-
DRIVERS-401 Prefer localhost for mongos load balancing
-
- Closed
-
- is related to
-
PYTHON-484 __pick_nearest doesn't work well for mongos seed list on local network
-
- Closed
-