[SERVER-5406] mongos does not allow new connections for 30 seconds->a couple minutes when an rs member goes down Created: 26/Mar/12  Updated: 11/Jul/16  Resolved: 18/Apr/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.0.3
Fix Version/s: 2.0.5, 2.1.1

Type: Bug Priority: Major - P3
Reporter: Kristina Chodorow (Inactive) Assignee: Greg Studer
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-5322 better messaging when cannot connect ... Closed
Operating System: ALL
Participants:

 Description   

See log @ 9:25:14, connecting to mongos takes ~3 minutes.

https://groups.google.com/d/msg/mongodb-user/HZV2I2GI2HY/xaD3hTJkMDwJ

https://groups.google.com/d/msg/mongodb-user/HZV2I2GI2HY/Ie3UopVe7KIJ



 Comments   
Comment by auto [ 18/Apr/12 ]

Author:

{u'login': u'gregstuder', u'name': u'gregs', u'email': u'greg@10gen.com'}

Message: SERVER-5406 allow slaveok auth on shard with normal timeout
Branch: v2.0
https://github.com/mongodb/mongo/commit/05b07daf5be2a90b70aec40328d8c3b20e60d68b

Comment by Greg Studer [ 18/Apr/12 ]

Resolving issue as fixed - future enhancements tracked for now via SERVER-5322

Comment by Vasyl [ 18/Apr/12 ]

Great! Thanks team!
We will upgrade mongoses to 2.0.5


Vasyl

Comment by Greg Studer [ 10/Apr/12 ]

The codereview site is internal, but the patch is now available here:

https://github.com/mongodb/mongo/commit/16b12c311f8bd5356910d2d70f8cc33e8c1904bb

> patch will not cause system downtime (we are on production now)
the patch should only affect mongos, so you can upgrade app-server by app-server

> patch will not mess up other things
It's a very limited change - so it's safe to use. Nothing wrt how data is routed or stored is touched.

> patching will not affect future stable mongo upgrades
It will potentially, if the patch isn't integrated into the next stable branch 2.0.x release. There's a pretty good chance it will though, I think. If you need the patch ASAP, you can build it pretty easily on a spare linux machine by using these steps :

  • git clone git://github.com/mongodb/mongo.git
  • git checkout r2.0.4 (or another version)
  • apply patch - can download and git apply or just do manually, as it's very small.
  • get build prerequisites - http://www.mongodb.org/display/DOCS/Building
  • build mongodb - scons mongos --release (the patch only affects mongos)
  • use new mongos to replace old mongoses - can replace one-by-one
Comment by Vasyl [ 10/Apr/12 ]

Proposed patch.
http://codereview.10gen.com/7683009/

Hi Greg!

Have a few questions >>

What are the u/p to access this patch?
Could you provide me with direct patching instruction? (i don't see such on mongo docs)
I need to be sure in following things before patching:

  • patch will not cause system downtime (we are on production now)
  • patch will not mess up other things
  • patching will not affect future stable mongo upgrades

thanks
vasyl

Comment by auto [ 06/Apr/12 ]

Author:

{u'login': u'gregstuder', u'name': u'gregs', u'email': u'greg@10gen.com'}

Message: SERVER-5406 allow slaveok auth on shard with normal timeout
Branch: master
https://github.com/mongodb/mongo/commit/16b12c311f8bd5356910d2d70f8cc33e8c1904bb

Comment by Greg Studer [ 05/Apr/12 ]

> 1) A blocked - minutes of timeouting (same as described in bug) until mongos is restarted. after restaring - mongos doesn't come up
> 2) B blocked - minutes of timeouting (same as described in bug) until mongos is restarted. after restaring - mongos doesn't come up

The patch should allow authentication via a secondary if one is up, and limits waiting 30s on the primary (and secondaries) if no secondary can be found. Issue when restarting mongos (from what I've seen) is that mongos actually comes up, but connecting via mongo shell throws a socket exception b/c to authenticate, we need a conn to the replica set. To establish the initial connection from a new mongos there needs to be a primary to ensure we're connecting to the right nodes (otherwise we might connect to an old stale node elsewhere and never know). Not sure if you're seeing something different here though.

> 3) Ar blocked - every 9th request (i.e. on 9,18,27 etc seconds) timeouting up to 5 seconds until mongos is restarted. after restaring - mongos doesn't come up
> 4) Br blocked - every 9th request (i.e. on 9,18,27 etc seconds) timeouting up to 10 seconds until mongos is restarted. after restarting - mongos doesnt come up

The issue here looks like it could be the replica set monitor thread - we'd need to correlate the logs with the delays when taking down the secondary. It pops up every 10s and uses 5s timeouts in checks on the replica nodes. I'm not sure why you weren't/aren't able to restart though, from my testing of blackholing the secondaries, mongos restarts ok and we can easily authenticate.

Comment by Greg Studer [ 05/Apr/12 ]

Proposed patch.

http://codereview.10gen.com/7683009/

Comment by Vasyl [ 28/Mar/12 ]

Thanks a lot for cooperation, Greg
I really appreciate it.
Will stay on hold for an update.

Vasyl

Comment by Greg Studer [ 28/Mar/12 ]

Thanks for this testing - it's extremely helpful in that it lets us know that sharded authentication is causing the problem. We've tracked down an issue which could cause problems like this authenticating when a node goes down, but a fix will require code changes, there's no workaround aside from stopping the mongo shell and retrying. Working on a patch now.

Comment by Vasyl [ 28/Mar/12 ]

Few more interesting tests. Hope this helps.

I got user-level and key-level authentication back. It was time-consuming a bit to on/off this everywhere (:
And tried to block each cluster node one by one on firewall level with full packets drop sourced by mongos

Quick recap of topology design

repSet1 [A,Ar,Aa] SHARD1 
repSet2 [B,Br,Ba] SHARD2 
 
A,B - primary nodes
Ar,Br - replicated nodes
Aa,Ba - arbiters
 
3 config servers
mongos running on application side
 
version 2.0.3 everywhere

1) A blocked - minutes of timeouting (same as described in bug) until mongos is restarted. after restaring - mongos doesn't come up
2) B blocked - minutes of timeouting (same as described in bug) until mongos is restarted. after restaring - mongos doesn't come up
3) Ar blocked - every 9th request (i.e. on 9,18,27 etc seconds) timeouting up to 5 seconds until mongos is restarted. after restaring - mongos doesn't come up
4) Br blocked - every 9th request (i.e. on 9,18,27 etc seconds) timeouting up to 10 seconds until mongos is restarted. after restarting - mongos doesnt come up

1) and 2) results same as in bug description links
3) and 4) results below

3) ##################################

-----------------------------
2012-03-28 13:22:43:503739668
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m4.497s
user    0m0.046s
sys     0m0.008s
2012-03-28 13:22:48:002989210
-----------------------------
2012-03-28 13:22:49:006779695
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.129s
user    0m0.051s
sys     0m0.005s
2012-03-28 13:22:49:137875296
-----------------------------
2012-03-28 13:22:50:142639085
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.246s
user    0m0.053s
sys     0m0.006s
2012-03-28 13:22:50:390917504
-----------------------------
2012-03-28 13:22:51:395146644
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.131s
user    0m0.048s
sys     0m0.005s
2012-03-28 13:22:51:528321721
-----------------------------
2012-03-28 13:22:52:536768287
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.242s
user    0m0.049s
sys     0m0.004s
2012-03-28 13:22:52:780979840
-----------------------------
2012-03-28 13:22:53:785824507
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.127s
user    0m0.046s
sys     0m0.007s
2012-03-28 13:22:53:915111082
-----------------------------
2012-03-28 13:22:54:937552849
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.249s
user    0m0.051s
sys     0m0.005s
2012-03-28 13:22:55:187990915
-----------------------------
2012-03-28 13:22:56:193061223
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.162s
user    0m0.046s
sys     0m0.007s
2012-03-28 13:22:56:356504802
-----------------------------
2012-03-28 13:22:57:360215965
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.229s
user    0m0.043s
sys     0m0.007s
2012-03-28 13:22:57:591209841
-----------------------------
2012-03-28 13:22:58:595388160
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m4.435s
user    0m0.048s
sys     0m0.004s
2012-03-28 13:23:03:032133665
-----------------------------
2012-03-28 13:23:04:037680590
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.130s
user    0m0.046s
sys     0m0.007s
2012-03-28 13:23:04:169271077
-----------------------------
2012-03-28 13:23:05:201017237
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.247s
user    0m0.045s
sys     0m0.008s
2012-03-28 13:23:05:450210914
-----------------------------
2012-03-28 13:23:06:454014186
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.140s
user    0m0.048s
sys     0m0.005s
2012-03-28 13:23:06:595807975
-----------------------------
2012-03-28 13:23:07:600326373
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.251s
user    0m0.051s
sys     0m0.005s
2012-03-28 13:23:07:853522751
-----------------------------
2012-03-28 13:23:08:858526220
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.154s
user    0m0.047s
sys     0m0.007s
2012-03-28 13:23:09:014727749
-----------------------------
2012-03-28 13:23:10:024144926
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.227s
user    0m0.050s
sys     0m0.008s
2012-03-28 13:23:10:252723250
-----------------------------
2012-03-28 13:23:11:256702926
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.127s
user    0m0.049s
sys     0m0.008s
2012-03-28 13:23:11:385127838
-----------------------------
2012-03-28 13:23:12:389117086
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.235s
user    0m0.051s
sys     0m0.006s
2012-03-28 13:23:12:625503593
-----------------------------
2012-03-28 13:23:13:629052723
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m4.444s
user    0m0.047s
sys     0m0.005s
2012-03-28 13:23:18:074713877
-----------------------------
2012-03-28 13:23:19:080269158
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.127s
user    0m0.046s
sys     0m0.008s
2012-03-28 13:23:19:209036203
-----------------------------

4) ################################################

-----------------------------
2012-03-28 13:27:30:537434735
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m9.396s
user    0m0.045s
sys     0m0.007s
2012-03-28 13:27:39:935795484
-----------------------------
2012-03-28 13:27:40:940492329
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.139s
user    0m0.045s
sys     0m0.008s
2012-03-28 13:27:41:081524780
-----------------------------
2012-03-28 13:27:42:186339463
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.150s
user    0m0.046s
sys     0m0.010s
2012-03-28 13:27:42:338184214
-----------------------------
2012-03-28 13:27:43:343970396
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.134s
user    0m0.048s
sys     0m0.004s
2012-03-28 13:27:43:479796058
-----------------------------
2012-03-28 13:27:44:621925094
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.254s
user    0m0.052s
sys     0m0.006s
2012-03-28 13:27:44:877681512
-----------------------------
2012-03-28 13:27:45:882449252
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.129s
user    0m0.044s
sys     0m0.009s
2012-03-28 13:27:46:013659350
-----------------------------
2012-03-28 13:27:47:018678180
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.237s
user    0m0.050s
sys     0m0.009s
2012-03-28 13:27:47:257855816
-----------------------------
2012-03-28 13:27:48:262780691
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.128s
user    0m0.050s
sys     0m0.005s
2012-03-28 13:27:48:392289983
-----------------------------
2012-03-28 13:27:49:401878766
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.246s
user    0m0.048s
sys     0m0.011s
2012-03-28 13:27:49:649599045
-----------------------------
2012-03-28 13:27:50:654008268
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m9.316s
user    0m0.049s
sys     0m0.006s
2012-03-28 13:27:59:971310822
-----------------------------
2012-03-28 13:28:00:976049869
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.124s
user    0m0.050s
sys     0m0.005s
2012-03-28 13:28:01:102234642
-----------------------------
2012-03-28 13:28:02:164744907
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.207s
user    0m0.046s
sys     0m0.007s
2012-03-28 13:28:02:374109724
-----------------------------
2012-03-28 13:28:03:378471110
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.129s
user    0m0.048s
sys     0m0.005s
2012-03-28 13:28:03:509703560
-----------------------------
2012-03-28 13:28:04:652515097
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.251s
user    0m0.051s
sys     0m0.007s
2012-03-28 13:28:04:905144230
-----------------------------
2012-03-28 13:28:05:909927519
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.136s
user    0m0.048s
sys     0m0.004s
2012-03-28 13:28:06:047560635
-----------------------------
2012-03-28 13:28:07:053145864
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.259s
user    0m0.047s
sys     0m0.010s
2012-03-28 13:28:07:314354193
-----------------------------
2012-03-28 13:28:08:318509714
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.137s
user    0m0.045s
sys     0m0.010s
2012-03-28 13:28:08:457413618
-----------------------------
2012-03-28 13:28:09:461832893
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.252s
user    0m0.046s
sys     0m0.007s
2012-03-28 13:28:09:716301846
-----------------------------
2012-03-28 13:28:10:720534660
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m9.292s
user    0m0.047s
sys     0m0.007s
2012-03-28 13:28:20:014182903
-----------------------------
2012-03-28 13:28:21:018762573
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.132s
user    0m0.047s
sys     0m0.006s
2012-03-28 13:28:21:153153956
-----------------------------

3), 4) ############# mongos doesn't come up

[root@mongos-server ~]# ps -aux | grep mongos
root      5035  0.0  0.0   4004   688 pts/6    R+   09:05   0:00 grep mongos
[root@mongos-server ~]#

-----------------------------
2012-03-29 08:58:42:647479332
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
Thu Mar 29 08:58:42 Error: couldn't connect to server localhost:27123 shell/mongo.js:86
exception: connect failed
 
real    0m0.052s
user    0m0.046s
sys     0m0.005s
-----------------------------
2012-03-29 08:58:43:704804905
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
Thu Mar 29 08:58:43 Error: couldn't connect to server localhost:27123 shell/mongo.js:86
exception: connect failed
 
real    0m0.050s
user    0m0.042s
sys     0m0.008s
-----------------------------
2012-03-29 08:58:44:759794499
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
Thu Mar 29 08:58:44 Error: couldn't connect to server localhost:27123 shell/mongo.js:86
exception: connect failed
 
real    0m0.054s
user    0m0.045s
sys     0m0.006s
-----------------------------

Thanks
Vasyl

Comment by Vasyl [ 28/Mar/12 ]

hello guys!

thanks you are on this! i really appreciate it.

just to be on the same page - short recap about previous testing with auth on, when nodeA was down
(more info in logs https://groups.google.com/forum/?fromgroups#!topic/mongodb-user/HZV2I2GI2HY)
system did authenticate succesfully to nodeB while nodeA wasn't been backuped yet. just with large timeouting
look at below time frames:

2012-03-23 09:25:16:479070162 64 bytes from nodeA (12.34.56.78): icmp_seq=123 ttl=60 time=2.80 ms
----------------- REBOOT ------------------------------------------------------------------------
2012-03-23 09:33:00:484940470 64 bytes from nodeA (12.34.56.78): icmp_seq=587 ttl=60 time=2.62 ms

2012-03-23 09:25:15:932604215
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    3m12.295s
user    0m0.049s
sys     0m0.004s
2012-03-23 09:28:28:229307913
-----------------------------
2012-03-23 09:28:29:233631999
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    3m39.289s
user    0m0.046s
sys     0m0.007s
2012-03-23 09:32:08:524013516
-----------------------------
2012-03-23 09:32:09:528561932
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m52.669s
user    0m0.043s
sys     0m0.010s
2012-03-23 09:33:02:199118681
-----------------------------
2012-03-23 09:33:03:203033931
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real    0m0.130s
user    0m0.045s
sys     0m0.008s
2012-03-23 09:33:03:334896508

can u pls update me on what type of auth did u mean here - user-level auth or key-level?
i turned off both just in case and system seems to be ok
but this is really bad approach to get things run
we can't just turn the authentication off, because cluster can be compromised from dozens of localhosted mongos in this case
below is ping and mongos-connection outputs

since its working fine without authentication i'm not sharing -vvvvvvv mongos log with --queit option off - it counts more then 100,000 lines during 1 hour testing and lots of info should be changed in this log for lots of servers (keyfiles auth hashes, usernames, server names, ips, ports, replicasets etc)
hope this is ok
thanks again for your help

[root@mongos-server ~]# ping nodeA | xargs -n1 -i bash -c 'echo `date +"%Y-%m-%d %H:%M:%S:%N"`" {}"' | tee ping_output
 
2012-03-28 08:27:28:179495676 64 bytes from nodeA (12.34.56.78): icmp_seq=1 ttl=60 time=2.62 ms
2012-03-28 08:27:29:175771854 64 bytes from nodeA (12.34.56.78): icmp_seq=2 ttl=60 time=2.78 ms
2012-03-28 08:27:30:225797724 64 bytes from nodeA (12.34.56.78): icmp_seq=3 ttl=60 time=2.82 ms
2012-03-28 08:27:31:178479743 64 bytes from nodeA (12.34.56.78): icmp_seq=4 ttl=60 time=2.77 ms
2012-03-28 08:27:32:178313105 64 bytes from nodeA (12.34.56.78): icmp_seq=5 ttl=60 time=2.65 ms
2012-03-28 08:27:33:178375236 64 bytes from nodeA (12.34.56.78): icmp_seq=6 ttl=60 time=2.69 ms
2012-03-28 08:27:34:178875568 64 bytes from nodeA (12.34.56.78): icmp_seq=7 ttl=60 time=2.74 ms
2012-03-28 08:27:35:179602472 64 bytes from nodeA (12.34.56.78): icmp_seq=8 ttl=60 time=2.68 ms
2012-03-28 08:27:36:181040721 64 bytes from nodeA (12.34.56.78): icmp_seq=9 ttl=60 time=2.81 ms
2012-03-28 08:27:37:182935388 64 bytes from nodeA (12.34.56.78): icmp_seq=10 ttl=60 time=2.84 ms
2012-03-28 08:27:38:181225925 64 bytes from nodeA (12.34.56.78): icmp_seq=11 ttl=60 time=2.68 ms
2012-03-28 08:27:39:182300305 64 bytes from nodeA (12.34.56.78): icmp_seq=12 ttl=60 time=2.60 ms
2012-03-28 08:27:40:192004641 64 bytes from nodeA (12.34.56.78): icmp_seq=13 ttl=60 time=2.65 ms
2012-03-28 08:27:41:184122332 64 bytes from nodeA (12.34.56.78): icmp_seq=14 ttl=60 time=2.72 ms
2012-03-28 08:27:42:183928042 64 bytes from nodeA (12.34.56.78): icmp_seq=15 ttl=60 time=2.66 ms
2012-03-28 08:27:43:184024252 64 bytes from nodeA (12.34.56.78): icmp_seq=16 ttl=60 time=2.69 ms
2012-03-28 08:27:44:267888105 64 bytes from nodeA (12.34.56.78): icmp_seq=17 ttl=60 time=2.64 ms
2012-03-28 08:27:45:186461731 64 bytes from nodeA (12.34.56.78): icmp_seq=18 ttl=60 time=2.62 ms
2012-03-28 08:27:46:187521023 64 bytes from nodeA (12.34.56.78): icmp_seq=19 ttl=60 time=3.56 ms
2012-03-28 08:27:47:187173723 64 bytes from nodeA (12.34.56.78): icmp_seq=20 ttl=60 time=3.01 ms
2012-03-28 08:27:48:187032041 64 bytes from nodeA (12.34.56.78): icmp_seq=21 ttl=60 time=2.62 ms
2012-03-28 08:27:49:188628696 64 bytes from nodeA (12.34.56.78): icmp_seq=22 ttl=60 time=3.08 ms
2012-03-28 08:27:50:188370128 64 bytes from nodeA (12.34.56.78): icmp_seq=23 ttl=60 time=2.64 ms
2012-03-28 08:27:51:288730267 64 bytes from nodeA (12.34.56.78): icmp_seq=24 ttl=60 time=2.82 ms
2012-03-28 08:27:52:189156720 64 bytes from nodeA (12.34.56.78): icmp_seq=25 ttl=60 time=3.08 ms
2012-03-28 08:27:53:189312464 64 bytes from nodeA (12.34.56.78): icmp_seq=26 ttl=60 time=2.73 ms
2012-03-28 08:27:54:190052479 64 bytes from nodeA (12.34.56.78): icmp_seq=27 ttl=60 time=2.66 ms
2012-03-28 08:27:55:201868602 64 bytes from nodeA (12.34.56.78): icmp_seq=28 ttl=60 time=2.64 ms
2012-03-28 08:27:56:191438065 64 bytes from nodeA (12.34.56.78): icmp_seq=29 ttl=60 time=2.67 ms
2012-03-28 08:27:57:191721798 64 bytes from nodeA (12.34.56.78): icmp_seq=30 ttl=60 time=2.79 ms
---------- SERVER IS DOWN FOR 1 HOUR -------------------------------------------------------------
2012-03-28 09:40:25:597389320 64 bytes from nodeA (12.34.56.78): icmp_seq=4376 ttl=60 time=2.72 ms
2012-03-28 09:40:26:518631480 64 bytes from nodeA (12.34.56.78): icmp_seq=4377 ttl=60 time=2.64 ms
2012-03-28 09:40:27:519826124 64 bytes from nodeA (12.34.56.78): icmp_seq=4378 ttl=60 time=2.71 ms
2012-03-28 09:40:28:520044852 64 bytes from nodeA (12.34.56.78): icmp_seq=4379 ttl=60 time=2.72 ms
2012-03-28 09:40:29:520155716 64 bytes from nodeA (12.34.56.78): icmp_seq=4380 ttl=60 time=2.63 ms
2012-03-28 09:40:30:521696894 64 bytes from nodeA (12.34.56.78): icmp_seq=4381 ttl=60 time=2.66 ms
2012-03-28 09:40:31:522842924 64 bytes from nodeA (12.34.56.78): icmp_seq=4382 ttl=60 time=2.76 ms
2012-03-28 09:40:32:551746696 64 bytes from nodeA (12.34.56.78): icmp_seq=4383 ttl=60 time=2.81 ms
2012-03-28 09:40:33:524139456 64 bytes from nodeA (12.34.56.78): icmp_seq=4384 ttl=60 time=2.75 ms
2012-03-28 09:40:34:536952832 64 bytes from nodeA (12.34.56.78): icmp_seq=4385 ttl=60 time=2.69 ms
2012-03-28 09:40:35:525727707 64 bytes from nodeA (12.34.56.78): icmp_seq=4386 ttl=60 time=2.87 ms
2012-03-28 09:40:36:526695751 64 bytes from nodeA (12.34.56.78): icmp_seq=4387 ttl=60 time=2.96 ms
2012-03-28 09:40:37:525856726 64 bytes from nodeA (12.34.56.78): icmp_seq=4388 ttl=60 time=2.79 ms
2012-03-28 09:40:38:526261185 64 bytes from nodeA (12.34.56.78): icmp_seq=4389 ttl=60 time=2.64 ms
2012-03-28 09:40:39:654854949 64 bytes from nodeA (12.34.56.78): icmp_seq=4390 ttl=60 time=2.78 ms
2012-03-28 09:40:40:634113449 64 bytes from nodeA (12.34.56.78): icmp_seq=4391 ttl=60 time=2.91 ms
2012-03-28 09:40:41:527573930 64 bytes from nodeA (12.34.56.78): icmp_seq=4392 ttl=60 time=2.77 ms
2012-03-28 09:40:42:527331963 64 bytes from nodeA (12.34.56.78): icmp_seq=4393 ttl=60 time=2.52 ms
2012-03-28 09:40:43:527848150 64 bytes from nodeA (12.34.56.78): icmp_seq=4394 ttl=60 time=2.71 ms
2012-03-28 09:40:44:600337078 64 bytes from nodeA (12.34.56.78): icmp_seq=4395 ttl=60 time=3.02 ms
2012-03-28 09:40:45:529159253 64 bytes from nodeA (12.34.56.78): icmp_seq=4396 ttl=60 time=2.83 ms
2012-03-28 09:40:46:530452248 64 bytes from nodeA (12.34.56.78): icmp_seq=4397 ttl=60 time=2.90 ms
2012-03-28 09:40:47:537905288 64 bytes from nodeA (12.34.56.78): icmp_seq=4398 ttl=60 time=2.79 ms
2012-03-28 09:40:48:531494411 64 bytes from nodeA (12.34.56.78): icmp_seq=4399 ttl=60 time=2.59 ms
2012-03-28 09:40:49:532724026 64 bytes from nodeA (12.34.56.78): icmp_seq=4400 ttl=60 time=2.59 ms
2012-03-28 09:40:50:534287048 64 bytes from nodeA (12.34.56.78): icmp_seq=4401 ttl=60 time=2.63 ms
2012-03-28 09:40:51:545614656 64 bytes from nodeA (12.34.56.78): icmp_seq=4402 ttl=60 time=2.69 ms
2012-03-28 09:40:52:534404843 64 bytes from nodeA (12.34.56.78): icmp_seq=4403 ttl=60 time=2.79 ms
2012-03-28 09:40:53:544563484 64 bytes from nodeA (12.34.56.78): icmp_seq=4404 ttl=60 time=3.02 ms
2012-03-28 09:40:54:535788056 64 bytes from nodeA (12.34.56.78): icmp_seq=4405 ttl=60 time=2.76 ms

[root@mongos-server ~]# while [ true ]; do date +"%Y-%m-%d %H:%M:%S:%N" && time mongo localhost:27123/db --eval "db.collection.count()" && date +"%Y-%m-%d %H:%M:%S:%N"; echo '-----------------------------'; sleep 1; done  2>&1 | tee mongos_output
 
2012-03-28 08:27:41:801915045
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.167s
user	0m0.047s
sys	0m0.008s
2012-03-28 08:27:41:970285940
-----------------------------
2012-03-28 08:27:42:974746890
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.049s
sys	0m0.008s
2012-03-28 08:27:43:043618854
-----------------------------
2012-03-28 08:27:44:120061373
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.169s
user	0m0.054s
sys	0m0.005s
2012-03-28 08:27:44:291020166
-----------------------------
2012-03-28 08:27:45:296262091
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.069s
user	0m0.044s
sys	0m0.012s
2012-03-28 08:27:45:367271505
-----------------------------
2012-03-28 08:27:46:440411972
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.221s
user	0m0.050s
sys	0m0.005s
2012-03-28 08:27:46:663159734
-----------------------------
2012-03-28 08:27:47:667529897
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.045s
sys	0m0.007s
2012-03-28 08:27:47:733565548
-----------------------------
2012-03-28 08:27:48:752294009
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.052s
sys	0m0.008s
2012-03-28 08:27:48:916046581
-----------------------------
2012-03-28 08:27:49:920951087
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.049s
sys	0m0.005s
2012-03-28 08:27:49:987031018
-----------------------------
2012-03-28 08:27:51:177596115
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.127s
user	0m0.047s
sys	0m0.007s
2012-03-28 08:27:51:306151177
-----------------------------
2012-03-28 08:27:52:309310883
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.050s
sys	0m0.004s
2012-03-28 08:27:52:375465578
-----------------------------
2012-03-28 08:27:53:389730260
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.049s
sys	0m0.010s
2012-03-28 08:27:53:553052809
-----------------------------
2012-03-28 08:27:54:556747614
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.045s
sys	0m0.007s
2012-03-28 08:27:54:622675425
-----------------------------
2012-03-28 08:27:55:706742370
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.046s
sys	0m0.006s
2012-03-28 08:27:55:870644181
 
------------------------------ SHUTDOWN STARTS ----------------------------
 
2012-03-28 08:27:56:875186725
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
Wed Mar 28 08:27:56 uncaught exception: error { "$err" : "socket exception", "code" : 11002 }
exception: login failed
 
real	0m0.057s
user	0m0.044s
sys	0m0.006s
-----------------------------
2012-03-28 08:27:58:022616633
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.177s
user	0m0.050s
sys	0m0.011s
2012-03-28 08:27:58:201298319
-----------------------------
2012-03-28 08:27:59:205639208
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.047s
sys	0m0.008s
2012-03-28 08:27:59:271610915
-----------------------------
2012-03-28 08:28:00:365865895
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.220s
user	0m0.047s
sys	0m0.008s
2012-03-28 08:28:00:587138608
-----------------------------
2012-03-28 08:28:01:591040558
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.047s
sys	0m0.005s
2012-03-28 08:28:01:656834185
-----------------------------
2012-03-28 08:28:02:678770899
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.161s
user	0m0.046s
sys	0m0.007s
2012-03-28 08:28:02:842173024
-----------------------------
2012-03-28 08:28:03:846454809
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.045s
sys	0m0.008s
2012-03-28 08:28:03:912450583
-----------------------------
2012-03-28 08:28:04:994820943
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.166s
user	0m0.046s
sys	0m0.012s
2012-03-28 08:28:05:162183093
-----------------------------
2012-03-28 08:28:06:165872328
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.050s
sys	0m0.004s
2012-03-28 08:28:06:232026777
-----------------------------
2012-03-28 08:28:07:310902710
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.161s
user	0m0.045s
sys	0m0.013s
2012-03-28 08:28:07:473695079
-----------------------------
2012-03-28 08:28:08:478292968
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.045s
sys	0m0.008s
2012-03-28 08:28:08:544862425
-----------------------------
2012-03-28 08:28:09:629542096
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.161s
user	0m0.049s
sys	0m0.006s
2012-03-28 08:28:09:792307786
-----------------------------
2012-03-28 08:28:10:796602246
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.044s
sys	0m0.009s
2012-03-28 08:28:10:862560944
-----------------------------
2012-03-28 08:28:11:968625279
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.229s
user	0m0.050s
sys	0m0.009s
2012-03-28 08:28:12:199973763
-----------------------------
2012-03-28 08:28:13:205106921
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.045s
sys	0m0.009s
2012-03-28 08:28:13:270478179
-----------------------------
2012-03-28 08:28:14:292401110
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.163s
user	0m0.051s
sys	0m0.007s
2012-03-28 08:28:14:457174440
-----------------------------
2012-03-28 08:28:15:460516139
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.048s
sys	0m0.007s
2012-03-28 08:28:15:526933673
-----------------------------
2012-03-28 08:28:16:611338165
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.218s
user	0m0.050s
sys	0m0.008s
2012-03-28 08:28:16:831526683
-----------------------------
2012-03-28 08:28:17:836022840
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.046s
sys	0m0.008s
2012-03-28 08:28:17:903903682
-----------------------------
2012-03-28 08:28:18:925764629
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.052s
sys	0m0.007s
2012-03-28 08:28:19:089529568
-----------------------------
2012-03-28 08:28:20:094366819
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.045s
sys	0m0.007s
2012-03-28 08:28:20:160309836
-----------------------------
2012-03-28 08:28:21:245751622
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.052s
sys	0m0.007s
2012-03-28 08:28:21:409888385
-----------------------------
2012-03-28 08:28:22:414664920
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.044s
sys	0m0.009s
2012-03-28 08:28:22:480442774
-----------------------------
2012-03-28 08:28:23:562355962
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.052s
sys	0m0.006s
2012-03-28 08:28:23:725457179
-----------------------------
2012-03-28 08:28:24:729140760
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.047s
sys	0m0.007s
2012-03-28 08:28:24:795124616
-----------------------------
2012-03-28 08:28:25:881019601
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.221s
user	0m0.049s
sys	0m0.007s
2012-03-28 08:28:26:104276673
-----------------------------
2012-03-28 08:28:27:108591289
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.047s
sys	0m0.006s
2012-03-28 08:28:27:174460688
-----------------------------
2012-03-28 08:28:28:195502661
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.049s
sys	0m0.005s
2012-03-28 08:28:28:359180483
-----------------------------
2012-03-28 08:28:29:363006565
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.049s
sys	0m0.008s
2012-03-28 08:28:29:431749411
-----------------------------
2012-03-28 08:28:30:510926344
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.046s
sys	0m0.009s
2012-03-28 08:28:30:675050261
-----------------------------
2012-03-28 08:28:31:680272193
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.046s
sys	0m0.008s
2012-03-28 08:28:31:745896955
-----------------------------
2012-03-28 08:28:32:831292270
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.168s
user	0m0.052s
sys	0m0.006s
2012-03-28 08:28:33:000512305
-----------------------------
2012-03-28 08:28:34:004949590
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.048s
sys	0m0.005s
2012-03-28 08:28:34:071764181
-----------------------------
2012-03-28 08:28:35:144985185
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.160s
user	0m0.046s
sys	0m0.010s
2012-03-28 08:28:35:307064148
-----------------------------
2012-03-28 08:28:36:311113776
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.043s
sys	0m0.011s
2012-03-28 08:28:36:376553962
-----------------------------
2012-03-28 08:28:37:461338358
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.224s
user	0m0.048s
sys	0m0.009s
2012-03-28 08:28:37:687296240
-----------------------------
2012-03-28 08:28:38:690746948
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.046s
sys	0m0.010s
2012-03-28 08:28:38:756799465
-----------------------------
2012-03-28 08:28:39:777804645
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.164s
user	0m0.048s
sys	0m0.007s
2012-03-28 08:28:39:943213737
-----------------------------
2012-03-28 08:28:40:947006530
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.047s
sys	0m0.006s
2012-03-28 08:28:41:012903860
-----------------------------
2012-03-28 08:28:42:094459101
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.227s
user	0m0.053s
sys	0m0.005s
2012-03-28 08:28:42:323514803
-----------------------------
2012-03-28 08:28:43:327438582
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.044s
sys	0m0.008s
2012-03-28 08:28:43:393453079
-----------------------------
2012-03-28 08:28:44:412062928
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.161s
user	0m0.044s
sys	0m0.008s
2012-03-28 08:28:44:574588823
-----------------------------
2012-03-28 08:28:45:579828529
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.049s
sys	0m0.006s
2012-03-28 08:28:45:647860273
-----------------------------
2012-03-28 08:28:46:728799278
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.161s
user	0m0.047s
sys	0m0.009s
2012-03-28 08:28:46:891066335
-----------------------------
2012-03-28 08:28:47:895305386
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.049s
sys	0m0.005s
2012-03-28 08:28:47:961429894
-----------------------------
2012-03-28 08:28:49:044316846
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.166s
user	0m0.050s
sys	0m0.013s
2012-03-28 08:28:49:211705759
-----------------------------
2012-03-28 08:28:50:215812447
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.006s
2012-03-28 08:28:50:282325100
-----------------------------
2012-03-28 08:28:51:375792077
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.221s
user	0m0.049s
sys	0m0.007s
2012-03-28 08:28:51:598113943
-----------------------------
2012-03-28 08:28:52:600962826
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.049s
sys	0m0.007s
2012-03-28 08:28:52:667360473
-----------------------------
2012-03-28 08:28:53:692040701
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.161s
user	0m0.050s
sys	0m0.008s
2012-03-28 08:28:53:855116615
-----------------------------
2012-03-28 08:28:54:860516488
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.044s
sys	0m0.011s
2012-03-28 08:28:54:926445110
-----------------------------
2012-03-28 08:28:56:008306987
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.048s
sys	0m0.008s
2012-03-28 08:28:56:172046970
-----------------------------
2012-03-28 08:28:57:175928718
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.050s
sys	0m0.005s
2012-03-28 08:28:57:242703251
-----------------------------
2012-03-28 08:28:58:327017191
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.161s
user	0m0.048s
sys	0m0.010s
2012-03-28 08:28:58:490229681
-----------------------------
2012-03-28 08:28:59:495236270
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.048s
sys	0m0.007s
2012-03-28 08:28:59:563540827
-----------------------------
2012-03-28 08:29:00:641577394
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.161s
user	0m0.049s
sys	0m0.008s
2012-03-28 08:29:00:804613479
 
...........................
...........................
...........................
...........................
 
2012-03-28 09:39:35:339693719
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.048s
sys	0m0.006s
2012-03-28 09:39:35:407486963
-----------------------------
2012-03-28 09:39:36:412690707
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.049s
sys	0m0.005s
2012-03-28 09:39:36:479667424
-----------------------------
2012-03-28 09:39:37:484892701
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.048s
sys	0m0.007s
2012-03-28 09:39:37:553479786
-----------------------------
2012-03-28 09:39:38:558016919
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.044s
sys	0m0.008s
2012-03-28 09:39:38:625114134
-----------------------------
2012-03-28 09:39:39:630313882
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.047s
sys	0m0.006s
2012-03-28 09:39:39:698320380
-----------------------------
2012-03-28 09:39:40:703539732
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.047s
sys	0m0.006s
2012-03-28 09:39:40:772293476
-----------------------------
2012-03-28 09:39:41:776635221
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.047s
sys	0m0.006s
2012-03-28 09:39:41:845265427
-----------------------------
2012-03-28 09:39:42:849871970
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.050s
sys	0m0.004s
2012-03-28 09:39:42:916858590
-----------------------------
2012-03-28 09:39:43:963293033
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.082s
user	0m0.048s
sys	0m0.008s
2012-03-28 09:39:44:047289836
-----------------------------
2012-03-28 09:39:45:052251285
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.049s
sys	0m0.005s
2012-03-28 09:39:45:119695105
-----------------------------
2012-03-28 09:39:46:186293659
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.172s
user	0m0.045s
sys	0m0.007s
2012-03-28 09:39:46:360017019
-----------------------------
2012-03-28 09:39:47:364626018
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.006s
2012-03-28 09:39:47:431467455
-----------------------------
2012-03-28 09:39:48:504318979
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.286s
user	0m0.054s
sys	0m0.010s
2012-03-28 09:39:48:792041391
-----------------------------
2012-03-28 09:39:49:797083637
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.052s
sys	0m0.002s
2012-03-28 09:39:49:865156435
-----------------------------
2012-03-28 09:39:50:915442865
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.087s
user	0m0.046s
sys	0m0.007s
2012-03-28 09:39:51:004976358
-----------------------------
2012-03-28 09:39:52:009483022
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.047s
sys	0m0.006s
2012-03-28 09:39:52:078420885
-----------------------------
2012-03-28 09:39:53:137297139
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.163s
user	0m0.050s
sys	0m0.008s
2012-03-28 09:39:53:302178908
-----------------------------
2012-03-28 09:39:54:306831589
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.049s
sys	0m0.008s
2012-03-28 09:39:54:376164809
-----------------------------
2012-03-28 09:39:55:453612110
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.161s
user	0m0.047s
sys	0m0.010s
2012-03-28 09:39:55:617019045
-----------------------------
2012-03-28 09:39:56:621253683
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.047s
sys	0m0.006s
2012-03-28 09:39:56:687615610
-----------------------------
2012-03-28 09:39:57:770096374
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.163s
user	0m0.051s
sys	0m0.007s
2012-03-28 09:39:57:935331232
-----------------------------
2012-03-28 09:39:58:972948493
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.113s
user	0m0.044s
sys	0m0.010s
2012-03-28 09:39:59:088014072
-----------------------------
2012-03-28 09:40:00:093259238
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.163s
user	0m0.054s
sys	0m0.007s
2012-03-28 09:40:00:257798527
-----------------------------
2012-03-28 09:40:01:262125369
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.050s
sys	0m0.004s
2012-03-28 09:40:01:329585349
-----------------------------
2012-03-28 09:40:02:402144609
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.163s
user	0m0.049s
sys	0m0.009s
2012-03-28 09:40:02:566870427
-----------------------------
2012-03-28 09:40:03:572530345
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.046s
sys	0m0.008s
2012-03-28 09:40:03:638983514
-----------------------------
2012-03-28 09:40:04:721097980
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.048s
sys	0m0.009s
2012-03-28 09:40:04:884826595
-----------------------------
2012-03-28 09:40:05:889081792
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.006s
2012-03-28 09:40:05:955985688
-----------------------------
2012-03-28 09:40:07:037154070
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.167s
user	0m0.055s
sys	0m0.007s
2012-03-28 09:40:07:206161694
-----------------------------
2012-03-28 09:40:08:211341650
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.050s
sys	0m0.004s
2012-03-28 09:40:08:277999829
-----------------------------
2012-03-28 09:40:09:483609477
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.130s
user	0m0.050s
sys	0m0.005s
2012-03-28 09:40:09:615316306
-----------------------------
2012-03-28 09:40:10:620743207
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.045s
sys	0m0.008s
2012-03-28 09:40:10:687501019
-----------------------------
2012-03-28 09:40:11:819650318
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.077s
user	0m0.050s
sys	0m0.008s
2012-03-28 09:40:11:898977343
-----------------------------
2012-03-28 09:40:12:904117644
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.049s
sys	0m0.004s
2012-03-28 09:40:12:970534761
-----------------------------
2012-03-28 09:40:14:001246671
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.173s
user	0m0.049s
sys	0m0.008s
2012-03-28 09:40:14:176668820
-----------------------------
2012-03-28 09:40:15:181600732
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.049s
sys	0m0.006s
2012-03-28 09:40:15:250691856
-----------------------------
2012-03-28 09:40:16:325466382
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.048s
sys	0m0.007s
2012-03-28 09:40:16:489610170
-----------------------------
2012-03-28 09:40:17:495001338
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.045s
sys	0m0.009s
2012-03-28 09:40:17:561954544
-----------------------------
2012-03-28 09:40:18:642245288
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.048s
sys	0m0.010s
2012-03-28 09:40:18:806729973
-----------------------------
2012-03-28 09:40:19:812719240
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.069s
user	0m0.049s
sys	0m0.006s
2012-03-28 09:40:19:883398231
-----------------------------
2012-03-28 09:40:20:958899992
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.163s
user	0m0.048s
sys	0m0.010s
2012-03-28 09:40:21:124212995
-----------------------------
2012-03-28 09:40:22:129754930
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.050s
sys	0m0.004s
2012-03-28 09:40:22:196682542
-----------------------------
2012-03-28 09:40:23:277182597
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.165s
user	0m0.052s
sys	0m0.006s
2012-03-28 09:40:23:444156770
-----------------------------
2012-03-28 09:40:24:449181246
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.048s
sys	0m0.005s
2012-03-28 09:40:24:515799213
 
---------------------------SHUTDOWN END----------------------------
 
2012-03-28 09:40:25:599856160
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.051s
sys	0m0.008s
2012-03-28 09:40:25:763581109
-----------------------------
2012-03-28 09:40:26:768578281
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.048s
sys	0m0.005s
2012-03-28 09:40:26:835162495
-----------------------------
2012-03-28 09:40:27:912441472
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.049s
sys	0m0.007s
2012-03-28 09:40:28:075925457
-----------------------------
2012-03-28 09:40:29:080002924
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.046s
sys	0m0.008s
2012-03-28 09:40:29:147735468
-----------------------------
2012-03-28 09:40:30:232575502
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.180s
user	0m0.050s
sys	0m0.012s
2012-03-28 09:40:30:414484648
-----------------------------
2012-03-28 09:40:31:419385036
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.637s
user	0m0.047s
sys	0m0.008s
2012-03-28 09:40:32:058832438
-----------------------------
2012-03-28 09:40:33:062711103
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.048s
sys	0m0.005s
2012-03-28 09:40:33:129056783
-----------------------------
2012-03-28 09:40:34:133923180
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.045s
sys	0m0.006s
2012-03-28 09:40:34:200990028
-----------------------------
2012-03-28 09:40:35:206112689
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.048s
sys	0m0.007s
2012-03-28 09:40:35:272548673
-----------------------------
2012-03-28 09:40:36:278294827
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.050s
sys	0m0.003s
2012-03-28 09:40:36:344805692
-----------------------------
2012-03-28 09:40:37:351514359
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.005s
2012-03-28 09:40:37:418391379
-----------------------------
2012-03-28 09:40:38:423682900
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.008s
2012-03-28 09:40:38:490959238
-----------------------------
2012-03-28 09:40:39:506056105
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.170s
user	0m0.050s
sys	0m0.009s
2012-03-28 09:40:39:678139909
-----------------------------
2012-03-28 09:40:40:683032293
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
Wed Mar 28 09:40:40 uncaught exception: login failed
exception: login failed
 
real	0m0.060s
user	0m0.048s
sys	0m0.005s
-----------------------------
2012-03-28 09:40:41:826326312
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
Wed Mar 28 09:40:42 uncaught exception: login failed
exception: login failed
 
real	0m0.228s
user	0m0.046s
sys	0m0.011s
-----------------------------
2012-03-28 09:40:43:058457222
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
Wed Mar 28 09:40:43 uncaught exception: login failed
exception: login failed
 
real	0m0.063s
user	0m0.046s
sys	0m0.005s
-----------------------------
2012-03-28 09:40:44:126714472
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.101s
user	0m0.047s
sys	0m0.008s
2012-03-28 09:40:44:229052700
-----------------------------
2012-03-28 09:40:45:233950821
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.048s
sys	0m0.004s
2012-03-28 09:40:45:301310488
-----------------------------
2012-03-28 09:40:46:307076422
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.007s
2012-03-28 09:40:46:374232201
-----------------------------
2012-03-28 09:40:47:379253675
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.183s
user	0m0.047s
sys	0m0.005s
2012-03-28 09:40:47:564039414
-----------------------------
2012-03-28 09:40:48:568518716
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.046s
sys	0m0.006s
2012-03-28 09:40:48:636287792
-----------------------------
2012-03-28 09:40:49:641756285
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.051s
sys	0m0.004s
2012-03-28 09:40:49:710584232
-----------------------------
2012-03-28 09:40:50:715791977
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.191s
user	0m0.052s
sys	0m0.008s
2012-03-28 09:40:50:908866934
-----------------------------
2012-03-28 09:40:51:913039084
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.007s
2012-03-28 09:40:51:980776770
-----------------------------
2012-03-28 09:40:52:985347845
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.049s
sys	0m0.005s
2012-03-28 09:40:53:052196964
-----------------------------
2012-03-28 09:40:54:057485620
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.050s
sys	0m0.006s
2012-03-28 09:40:54:124700033
-----------------------------
2012-03-28 09:40:55:129640219
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.004s
2012-03-28 09:40:55:196459609
-----------------------------
2012-03-28 09:40:56:204847025
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.164s
user	0m0.053s
sys	0m0.006s
2012-03-28 09:40:56:370708794
-----------------------------
2012-03-28 09:40:57:376183197
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.048s
sys	0m0.006s
2012-03-28 09:40:57:442731901
-----------------------------
2012-03-28 09:40:58:521550752
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.169s
user	0m0.051s
sys	0m0.008s
2012-03-28 09:40:58:692596408
-----------------------------
2012-03-28 09:40:59:697568042
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.046s
sys	0m0.005s
2012-03-28 09:40:59:764119933
-----------------------------
2012-03-28 09:41:00:866182649
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.226s
user	0m0.048s
sys	0m0.011s
2012-03-28 09:41:01:094199497
-----------------------------
2012-03-28 09:41:02:098860400
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.007s
2012-03-28 09:41:02:165779086
-----------------------------
2012-03-28 09:41:03:184533722
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.051s
sys	0m0.006s
2012-03-28 09:41:03:348873063
-----------------------------
2012-03-28 09:41:04:354291162
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.048s
sys	0m0.009s
2012-03-28 09:41:04:423577562
-----------------------------
2012-03-28 09:41:05:500846693
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.170s
user	0m0.051s
sys	0m0.008s
2012-03-28 09:41:05:672849187
-----------------------------
2012-03-28 09:41:06:676696250
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.048s
sys	0m0.007s
2012-03-28 09:41:06:745535762
-----------------------------
2012-03-28 09:41:08:018595305
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.093s
user	0m0.050s
sys	0m0.009s
2012-03-28 09:41:08:114166339
-----------------------------
2012-03-28 09:41:09:119130691
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.069s
user	0m0.047s
sys	0m0.009s
2012-03-28 09:41:09:190769312
-----------------------------
2012-03-28 09:41:10:230741408
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.089s
user	0m0.049s
sys	0m0.007s
2012-03-28 09:41:10:321385007
-----------------------------
2012-03-28 09:41:11:326466600
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.125s
user	0m0.048s
sys	0m0.004s
2012-03-28 09:41:11:453024779
-----------------------------
2012-03-28 09:41:12:458078415
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.166s
user	0m0.045s
sys	0m0.007s
2012-03-28 09:41:12:626292422
-----------------------------
2012-03-28 09:41:13:631017963
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.044s
sys	0m0.007s
2012-03-28 09:41:13:697538468
-----------------------------
2012-03-28 09:41:14:768596815
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.176s
user	0m0.050s
sys	0m0.007s
2012-03-28 09:41:14:946622312
-----------------------------
2012-03-28 09:41:15:951441086
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.005s
2012-03-28 09:41:16:018238277
-----------------------------
2012-03-28 09:41:17:086787448
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.175s
user	0m0.050s
sys	0m0.006s
2012-03-28 09:41:17:263878941
-----------------------------
2012-03-28 09:41:18:270688729
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.049s
sys	0m0.008s
2012-03-28 09:41:18:340183754
-----------------------------
2012-03-28 09:41:19:406316429
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.168s
user	0m0.047s
sys	0m0.008s
2012-03-28 09:41:19:576822755
-----------------------------
2012-03-28 09:41:20:581096319
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.051s
sys	0m0.005s
2012-03-28 09:41:20:648252665
-----------------------------
2012-03-28 09:41:21:868006086
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.077s
user	0m0.051s
sys	0m0.007s
2012-03-28 09:41:21:947394270
-----------------------------
2012-03-28 09:41:22:952702925
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.048s
sys	0m0.008s
2012-03-28 09:41:23:022681285
-----------------------------
2012-03-28 09:41:24:040385524
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.163s
user	0m0.051s
sys	0m0.005s
2012-03-28 09:41:24:205301803
-----------------------------
2012-03-28 09:41:25:209974597
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.046s
sys	0m0.006s
2012-03-28 09:41:25:276703187
-----------------------------
2012-03-28 09:41:26:356408855
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.232s
user	0m0.047s
sys	0m0.010s
2012-03-28 09:41:26:590485711
-----------------------------
2012-03-28 09:41:27:595280132
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.064s
user	0m0.046s
sys	0m0.007s
2012-03-28 09:41:27:662006330
-----------------------------
2012-03-28 09:41:28:675731646
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.167s
user	0m0.052s
sys	0m0.007s
2012-03-28 09:41:28:845189023
-----------------------------
2012-03-28 09:41:29:849745627
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.047s
sys	0m0.005s
2012-03-28 09:41:29:917508112
-----------------------------
2012-03-28 09:41:31:000375599
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.163s
user	0m0.051s
sys	0m0.004s
2012-03-28 09:41:31:165600256
-----------------------------
2012-03-28 09:41:32:170260952
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.048s
sys	0m0.005s
2012-03-28 09:41:32:237080131
-----------------------------
2012-03-28 09:41:33:329552395
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.225s
user	0m0.050s
sys	0m0.006s
2012-03-28 09:41:33:556785934
-----------------------------
2012-03-28 09:41:34:561695400
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.070s
user	0m0.050s
sys	0m0.004s
2012-03-28 09:41:34:633689295
-----------------------------
2012-03-28 09:41:35:652280753
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.052s
sys	0m0.006s
2012-03-28 09:41:35:816753621
-----------------------------
2012-03-28 09:41:36:821981843
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.049s
sys	0m0.004s
2012-03-28 09:41:36:888543887
-----------------------------
2012-03-28 09:41:37:964212754
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.164s
user	0m0.048s
sys	0m0.006s
2012-03-28 09:41:38:130109708
-----------------------------
2012-03-28 09:41:39:134449925
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.049s
sys	0m0.005s
2012-03-28 09:41:39:201269736
-----------------------------
2012-03-28 09:41:40:280459961
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.162s
user	0m0.048s
sys	0m0.007s
2012-03-28 09:41:40:443941200
-----------------------------
2012-03-28 09:41:41:448862050
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.045s
sys	0m0.007s
2012-03-28 09:41:41:515932897
-----------------------------
2012-03-28 09:41:42:602027817
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.164s
user	0m0.050s
sys	0m0.007s
2012-03-28 09:41:42:767831302
-----------------------------
2012-03-28 09:41:43:772202219
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.050s
sys	0m0.003s
2012-03-28 09:41:43:839023443
-----------------------------
2012-03-28 09:41:44:923593614
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.165s
user	0m0.051s
sys	0m0.010s
2012-03-28 09:41:45:090515845
-----------------------------
2012-03-28 09:41:46:095791063
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.066s
user	0m0.049s
sys	0m0.005s
2012-03-28 09:41:46:164366341
-----------------------------
2012-03-28 09:41:47:245870978
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.222s
user	0m0.049s
sys	0m0.009s
2012-03-28 09:41:47:470179662
-----------------------------
2012-03-28 09:41:48:475075342
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.067s
user	0m0.047s
sys	0m0.007s
2012-03-28 09:41:48:543788963
-----------------------------
2012-03-28 09:41:49:562984906
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.165s
user	0m0.048s
sys	0m0.010s
2012-03-28 09:41:49:730363525
-----------------------------
2012-03-28 09:41:50:735429740
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.068s
user	0m0.049s
sys	0m0.007s
2012-03-28 09:41:50:805780651
-----------------------------
2012-03-28 09:41:51:883690497
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.220s
user	0m0.051s
sys	0m0.008s
2012-03-28 09:41:52:106025027
-----------------------------
2012-03-28 09:41:53:110901425
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.051s
sys	0m0.004s
2012-03-28 09:41:53:177871572
-----------------------------
2012-03-28 09:41:54:197598347
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.163s
user	0m0.047s
sys	0m0.009s
2012-03-28 09:41:54:362524211
-----------------------------
2012-03-28 09:41:55:367331535
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.047s
sys	0m0.005s
2012-03-28 09:41:55:434245676
-----------------------------
2012-03-28 09:41:56:513342006
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.168s
user	0m0.048s
sys	0m0.009s
2012-03-28 09:41:56:683447996
-----------------------------
2012-03-28 09:41:57:688639715
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.052s
sys	0m0.004s
2012-03-28 09:41:57:755907443
-----------------------------
2012-03-28 09:41:58:829845444
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.223s
user	0m0.047s
sys	0m0.009s
2012-03-28 09:41:59:055015235
-----------------------------
2012-03-28 09:42:00:060161013
MongoDB shell version: 2.0.3
connecting to: localhost:27123/db
2095
 
real	0m0.065s
user	0m0.048s
sys	0m0.004s
2012-03-28 09:42:00:126957309
-----------------------------

Vasyl

Comment by Vasyl [ 27/Mar/12 ]

ok deal

v

Comment by Randolph Tan [ 27/Mar/12 ]

It would also be very helpful to have mongos logs running on the highest verbosity you can afford (by passing -vvvvv, lesser v for lesser verbosity).

Comment by Vasyl [ 27/Mar/12 ]

i will retest this tomorrow without --quiet option and with authentication off
if you need anything else - let me know, b/c we are putting sharding cluster to production in few days - and thus will not be able to track this on our side.

thanks a lot
vasyl

Comment by Greg Studer [ 27/Mar/12 ]

A few questions - are you running mongos with --quiet on? We aren't seeing indications of new connections to the shell in the log, even when there are new connections being made, and this makes things difficult to track.

Is it possible to retry this without authentication - that's our current hypothesis. We're trying to reproduce on our side but it would be great if it was easy to eliminate/confirm that factor.

Generated at Thu Feb 08 03:08:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.