[SERVER-8024] getLastError >90 ms delay over mongos Created: 23/Dec/12  Updated: 25/Jun/15  Resolved: 23/Jul/14

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.0.6, 2.2.0, 2.2.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Konstantin Belov Assignee: Ramon Fernandez Marina
Resolution: Incomplete Votes: 0
Labels: freebsd, sharding
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

FreeBSD 8.2 / 9.0 / 9.1 PRERELEASE


Operating System: FreeBSD
Steps To Reproduce:

Simple php script:

<?php
// connecting to mongos
$m = new MongoClient('mongodb://127.0.0.1:20027');
$db = $m->selectDB('test');

$collection = new MongoCollection($db, 'c1');

$mt = microtime(true);
$r = $collection->insert(array('n' => 1));
echo round( (microtime(true) - $mt) * 1000), "\n";

Participants:

 Description   

I'am trying to deploy shard cluster for tests.
Using docs http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/

And every time when execute update there is strange delay 90+ ms.

mongos.log:

Sun Dec 23 23:41:06 [conn1] Request::process begin ns: test.$cmd msg id: 2 op: 2004 attempt: 0
Sun Dec 23 23:41:06 [conn1] single query: test.$cmd

{ getlasterror: 1 }

ntoreturn: -1 options : 0 0ms
Sun Dec 23 23:41:06 [conn1] gathering response for gle from: 127.0.0.1:20001
Sun Dec 23 23:41:06 [conn1] T1: 0ms
Sun Dec 23 23:41:06 [conn1] T2: 99ms
Sun Dec 23 23:41:06 [conn1] gathering writebacks from 3 hosts for gle (127.0.0.1:20001)
Sun Dec 23 23:41:06 [conn1] gathering writebacks for single-shard gle from: 127.0.0.1:20002
Sun Dec 23 23:41:06 [conn1] gathering writebacks for single-shard gle from: 127.0.0.1:20003
Sun Dec 23 23:41:06 [conn1] checking 0 writebacks for gle (127.0.0.1:20001)
Sun Dec 23 23:41:06 [conn1] Request::process end ns: test.$cmd msg id: 2 op: 2004 attempt: 0 99ms

I add some debug information into: src/mongo/s/client_info.cpp (function ClientInfo::getLastError):

Timer t;
LOG(5) << "gathering response for gle from: " << theShard << endl;

ShardConnection conn( theShard , "" );
try

{ LOG(5) << "T1: " << t.millis() << "ms" << endl; ok = conn->runCommand( dbName , options , res ); LOG(5) << "T2: " << t.millis() << "ms" << endl; }

 Comments   
Comment by Ramon Fernandez Marina [ 23/Jul/14 ]

belov, we haven't heard back from you for some time, so we're going to mark this ticket as resolved. If this is still an issue for you, feel free to reopen the ticket and provide further information.

Regards,
Ramón.

Comment by Ramon Fernandez Marina [ 14/Jul/14 ]

belov, can you please let us know whether you're still seeing the 90ms delay? If the answer is yes, is that on 2.0/2.2 or in a newer version?

Thanks,
Ramón.

Comment by Ramon Fernandez Marina [ 27/Jun/14 ]

Hi belov,

apologies for the late reply, seems we let this one fall through the cracks. Are you still on 2.0/2.2? If you have upgraded to 2.4/2.6, can you please let us know if you are still seeing this 90ms delay?

Comment by sam.helman@10gen.com [ 31/Jan/13 ]

Hello,

We have had some historical issues with freebsd and MongoDB related to performance - additionally, since there are relatively few freebsd deployments, it is easier to run into edge cases which have not yet been discovered. If general, we recommend not using freebsd for deployment. Is this a possibility for you?

There are some threads related to performance on freebsd linked here: http://serverfault.com/questions/401659/mongodb-on-freebsd.

Thanks

Comment by Konstantin Belov [ 25/Jan/13 ]

Sam, hello!

Please read my previous post. Under freebsd problem caused by socket implementation (mongo/util/net/sock.cpp)
We use amd64 kernel, may be we need to tune some sysctl net parameters.

About config:
We test two setups:
1. mongod, mongod+cfg and mongos on the same machine running under freebsd 8.2
2. mongos (the same setup) on another machine running under Ubuntu but connecting to mongod on the first machine.

If we make query direct to mongod (using mongo shell) or via linux mongos - delay is 0 ms. If we make query via freebsd mongos - delay is ALWAYS 90+ ms. I don't know why under freebsd when we read first four bytes from socket we get ~90 ms timeout

Comment by sam.helman@10gen.com [ 24/Jan/13 ]

Hello,

When you say that you tried on another machine under linux, do you mean that you tried the same setup on a different machine and had no issues? Are there additional processes (besides mongod) running on the slower machine that might be causing a general I/O slowdown?

Are there any operations besides inserts which are causing the 90ms delay?

Thanks

Comment by Konstantin Belov [ 24/Jan/13 ]

Hello again!

This timeout problem exists not only with command "getLastError", i add debug information into: "util/net/sock.cpp" at line 774 in function "unsafe_recv":

int Socket::unsafe_recv( char *buf, int max )

mongos.log:
...
Thu Jan 24 18:48:10 [Balancer] Socket::recv() len = 4
Thu Jan 24 18:48:10 [Balancer] Socket::recv(), unsafe_recv done in 90 ms
...
Thu Jan 24 18:50:01 [conn3] Socket::recv() len = 4
Thu Jan 24 18:50:01 [conn3] Socket::recv(), unsafe_recv done in 90 ms
Thu Jan 24 18:50:01 [conn3] Socket::recv() len = 97
Thu Jan 24 18:50:01 [conn3] Socket::recv(), unsafe_recv done in 0 ms
Thu Jan 24 18:50:01 [conn3] psock->recv()

So when we read first four bytes from socket we get ~90 ms timeout, why? o_O

Comment by Konstantin Belov [ 19/Jan/13 ]

Hello!

We use php driver 1.3.2, write concern = 1, in collections we have only one index - "_id".
But if we run mongos on another machine under linux (mongod still on freebsd) - there is no delay!!

Comment by sam.helman@10gen.com [ 16/Jan/13 ]

Hello,

The 90 ms delay could have a number of causes. If your insert is causing an index to be updated, this could create a delay since the index is updated on all shards. What write concern are you using, and which version of the php driver? Do you have any indexes on the collection?

Thank you

Generated at Thu Feb 08 03:16:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.