Started a server today and couldn't connect to the database:

$ mongo
MongoDB shell version: 1.8.2
connecting to: test
> use example
switched to db example
> show collections
Mon Jun 20 22:24:43 uncaught exception: error:

{ "$err" : "socket exception", "code" : 11002 }

This was in mongos.log:

Mon Jun 20 21:56:28 /usr/bin/mongos db version v1.8.2-rc3, pdfile version 4.5 starting (--help for usage)
Mon Jun 20 21:56:28 git version: 2d7719228787c9c8100456bc70bf860ec2885732
Mon Jun 20 21:56:28 build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Mon Jun 20 21:56:28 config string : mongodb-config03.example.com:27019,mongodb-config02.example.com:27019,mongodb-config01.example.com:27019
Mon Jun 20 21:56:28 creating new connection to:mongodb-config03.example.com:27019
Mon Jun 20 21:56:28 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:28 creating new connection to:mongodb-config02.example.com:27019
Mon Jun 20 21:56:28 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:28 creating new connection to:mongodb-config01.example.com:27019
Mon Jun 20 21:56:28 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:28 SyncClusterConnection connecting to [mongodb-config03.example.com:27019]
Mon Jun 20 21:56:28 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:28 SyncClusterConnection connecting to [mongodb-config02.example.com:27019]
Mon Jun 20 21:56:28 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:28 SyncClusterConnection connecting to [mongodb-config01.example.com:27019]
Mon Jun 20 21:56:28 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:28 MaxChunkSize: value: 200
Mon Jun 20 21:56:28 BackgroundJob starting: Balancer
Mon Jun 20 21:56:28 [Balancer] about to contact config servers and shards
Mon Jun 20 21:56:28 [websvr] web admin interface listening on port 28017
Mon Jun 20 21:56:28 [websvr] fd limit hard:65535 soft:65535 max conn: 52428
Mon Jun 20 21:56:28 [mongosMain] waiting for connections on port 27017
Mon Jun 20 21:56:28 [mongosMain] fd limit hard:65535 soft:65535 max conn: 52428
Mon Jun 20 21:56:28 BackgroundJob starting: cursorTimeout
Mon Jun 20 21:56:29 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:34 [Balancer] error connecting to seed mongodb04.example.com:27018: couldn't connect to server mongodb04.example.com:27018
Mon Jun 20 21:56:34 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:39 [Balancer] error connecting to seed mongodb05.example.com:27018: couldn't connect to server mongodb05.example.com:27018
Mon Jun 20 21:56:39 BackgroundJob starting: ReplicaSetMonitorWatcher
Mon Jun 20 21:56:39 [ReplicaSetMonitorWatcher] starting
Mon Jun 20 21:56:39 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:44 [Balancer] error connecting to seed mongodb02.example.com:27018: couldn't connect to server mongodb02.example.com:27018
Mon Jun 20 21:56:44 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:47 [Balancer] ReplicaSetMonitor::_checkConnection: mongodb01.example.com:27018

{ setName: "1", ismaster: true, secondary: false, hosts: [ "mongodb01.example.com:27018", "mongodb02.example.com:27018" ], arbiters: [ "mongodb03.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:56:47 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:47 [Balancer] updated set (1) to: 1/mongodb01.example.com:27018,mongodb02.example.com:27018
Mon Jun 20 21:56:47 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:47 [Balancer] ReplicaSetMonitor::_checkConnection: mongodb08.example.com:27018

{ setName: "3", ismaster: false, secondary: true, hosts: [ "mongodb08.example.com:27018", "mongodb07.example.com:27018" ], arbiters: [ "mongodb09.example.com:27018" ], primary: "mongodb07.example.com:27018", maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:56:47 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:47 [Balancer] updated set (3) to: 3/mongodb08.example.com:27018,mongodb07.example.com:27018
Mon Jun 20 21:56:48 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:48 [Balancer] ReplicaSetMonitor::_checkConnection: mongodb11.example.com:27018

{ setName: "4", ismaster: false, secondary: true, hosts: [ "mongodb11.example.com:27018", "mongodb10.example.com:27018" ], arbiters: [ "mongodb12.example.com:27018" ], primary: "mongodb10.example.com:27018", maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:56:48 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:48 [Balancer] updated set (4) to: 4/mongodb11.example.com:27018,mongodb10.example.com:27018
Mon Jun 20 21:56:48 [Balancer] _check : 1/mongodb01.example.com:27018,mongodb02.example.com:27018
Mon Jun 20 21:56:48 [Balancer] ReplicaSetMonitor::_checkConnection: mongodb01.example.com:27018

{ setName: "1", ismaster: true, secondary: false, hosts: [ "mongodb01.example.com:27018", "mongodb02.example.com:27018" ], arbiters: [ "mongodb03.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:56:48 BackgroundJob starting: ConnectBG
Mon Jun 20 21:56:48 [Balancer] _check : 2/
Mon Jun 20 21:56:50 [Balancer] User Assertion: 10009:ReplicaSetMonitor no master found for set: 2
Mon Jun 20 21:56:50 [Balancer] warning: could not initialize balancer, please check that all shards and config servers are up
Mon Jun 20 21:56:50 [Balancer] will retry to initialize balancer in one minute
Mon Jun 20 21:56:59 [ReplicaSetMonitorWatcher] checking replica set: 1
Mon Jun 20 21:56:59 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb01.example.com:27018

{ setName: "1", ismaster: true, secondary: false, hosts: [ "mongodb01.example.com:27018", "mongodb02.example.com:27018" ], arbiters: [ "mongodb03.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:56:59 [ReplicaSetMonitorWatcher] checking replica set: 2
Mon Jun 20 21:56:59 [ReplicaSetMonitorWatcher] _check : 2/
Mon Jun 20 21:57:01 [ReplicaSetMonitorWatcher] checking replica set: 3
Mon Jun 20 21:57:01 [ReplicaSetMonitorWatcher] _check : 3/mongodb08.example.com:27018,mongodb07.example.com:27018
Mon Jun 20 21:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb08.example.com:27018

{ setName: "3", ismaster: false, secondary: true, hosts: [ "mongodb08.example.com:27018", "mongodb07.example.com:27018" ], arbiters: [ "mongodb09.example.com:27018" ], primary: "mongodb07.example.com:27018", maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb07.example.com:27018

{ setName: "3", ismaster: true, secondary: false, hosts: [ "mongodb07.example.com:27018", "mongodb08.example.com:27018" ], arbiters: [ "mongodb09.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:57:01 [ReplicaSetMonitorWatcher] checking replica set: 4
Mon Jun 20 21:57:01 [ReplicaSetMonitorWatcher] _check : 4/mongodb11.example.com:27018,mongodb10.example.com:27018
Mon Jun 20 21:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb11.example.com:27018

{ setName: "4", ismaster: false, secondary: true, hosts: [ "mongodb11.example.com:27018", "mongodb10.example.com:27018" ], arbiters: [ "mongodb12.example.com:27018" ], primary: "mongodb10.example.com:27018", maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:57:01 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb10.example.com:27018

{ setName: "4", ismaster: true, secondary: false, hosts: [ "mongodb10.example.com:27018", "mongodb11.example.com:27018" ], arbiters: [ "mongodb12.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:57:21 [ReplicaSetMonitorWatcher] checking replica set: 1
Mon Jun 20 21:57:21 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb01.example.com:27018

{ setName: "1", ismaster: true, secondary: false, hosts: [ "mongodb01.example.com:27018", "mongodb02.example.com:27018" ], arbiters: [ "mongodb03.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:57:21 [ReplicaSetMonitorWatcher] checking replica set: 2
Mon Jun 20 21:57:21 [ReplicaSetMonitorWatcher] _check : 2/
Mon Jun 20 21:57:23 [ReplicaSetMonitorWatcher] checking replica set: 3
Mon Jun 20 21:57:23 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb07.example.com:27018

{ setName: "3", ismaster: true, secondary: false, hosts: [ "mongodb07.example.com:27018", "mongodb08.example.com:27018" ], arbiters: [ "mongodb09.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:57:23 [ReplicaSetMonitorWatcher] checking replica set: 4
Mon Jun 20 21:57:23 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb10.example.com:27018

{ setName: "4", ismaster: true, secondary: false, hosts: [ "mongodb10.example.com:27018", "mongodb11.example.com:27018" ], arbiters: [ "mongodb12.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:57:43 [ReplicaSetMonitorWatcher] checking replica set: 1
Mon Jun 20 21:57:43 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb01.example.com:27018

{ setName: "1", ismaster: true, secondary: false, hosts: [ "mongodb01.example.com:27018", "mongodb02.example.com:27018" ], arbiters: [ "mongodb03.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:57:43 [ReplicaSetMonitorWatcher] checking replica set: 2
Mon Jun 20 21:57:43 [ReplicaSetMonitorWatcher] _check : 2/
Mon Jun 20 21:57:45 [ReplicaSetMonitorWatcher] checking replica set: 3
Mon Jun 20 21:57:45 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb07.example.com:27018

{ setName: "3", ismaster: true, secondary: false, hosts: [ "mongodb07.example.com:27018", "mongodb08.example.com:27018" ], arbiters: [ "mongodb09.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Mon Jun 20 21:57:45 [ReplicaSetMonitorWatcher] checking replica set: 4
Mon Jun 20 21:57:45 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb10.example.com:27018

{ setName: "4", ismaster: true, secondary: false, hosts: [ "mongodb10.example.com:27018", "mongodb11.example.com:27018" ], arbiters: [ "mongodb12.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

Bouncing mongos fixed this.