Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: 1.7.4
Component/s: None
Labels:
None
Environment:
Linux 64-Bit

Operating System:
Linux
Confidence Status:
None
Work Order:
3

Aha! Reference:
None
Tracking Level:
None
Risk Status:
None
Exec Notes:
None
Goal Name:
None
Goal Link:
None

Having a 3 Sharded VM-Environment with each Shard having 3 Replicas w/o Arbiter (all rs.status() ok and looking good as well as db.printShardingStatus()), dropping a non-empty and indexed database via a local Routing-Server gives me the following errors.

I also had errors while finding documents via distinct, but the errors while dropping are repeatable, so I'm posting them.
Feel free to contact me for details or remote debugging. Or please give me a hint, how and why I have misconfigured something. Thank you!

MongoDB Console:

MongoDB shell version: 1.7.4
connecting to: webmanagement
> show dbs
admin (empty)
local (empty)
shop (empty)
webmanagement (empty)
> use shop
switched to db shop
> db.dropDatabase()

{ "dropped" : "shop", "ok" : 1 }

> show dbs
admin (empty)
local (empty)
webmanagement (empty)

Errors/Console-Log at Routing-Server:

Tue Jan 18 16:10:24 ./mongos db version v1.7.4, pdfile version 4.5 starting (--help for usage)
Tue Jan 18 16:10:24 git version: fecf14def3bd51b91e0af55d61e9de2b9f918dcf
Tue Jan 18 16:10:24 sys info: Darwin erh2.10gen.cc 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_40
Tue Jan 18 16:10:24 SyncClusterConnection connecting to [mongo-int1:9999]
Tue Jan 18 16:10:24 SyncClusterConnection connecting to [mongo-int2:9999]
Tue Jan 18 16:10:24 SyncClusterConnection connecting to [mongo-int3:9999]
Tue Jan 18 16:10:24 [mongosMain] waiting for connections on port 27017
Tue Jan 18 16:10:24 [Balancer] about to contact config servers and shards
Tue Jan 18 16:10:24 [websvr] web admin interface listening on port 28017
Tue Jan 18 16:10:24 [Balancer] config servers and shards contacted successfully
Tue Jan 18 16:10:24 [Balancer] balancer id: Gerhard-Balthasars-iMac.local:27017 started at Jan 18 16:10:24
Tue Jan 18 16:10:24 [LockPinger] SyncClusterConnection connecting to [mongo-int1:9999]
Tue Jan 18 16:10:24 [LockPinger] SyncClusterConnection connecting to [mongo-int2:9999]
Tue Jan 18 16:10:24 [Balancer] SyncClusterConnection connecting to [mongo-int1:9999]
Tue Jan 18 16:10:24 [LockPinger] SyncClusterConnection connecting to [mongo-int3:9999]
Tue Jan 18 16:10:24 [Balancer] SyncClusterConnection connecting to [mongo-int2:9999]
Tue Jan 18 16:10:24 [Balancer] SyncClusterConnection connecting to [mongo-int3:9999]
Tue Jan 18 16:10:29 [mongosMain] connection accepted from 127.0.0.1:59380 #1
Tue Jan 18 16:10:29 [conn1] end connection 127.0.0.1:59380
Tue Jan 18 16:10:32 [mongosMain] connection accepted from 127.0.0.1:59381 #2
Tue Jan 18 16:10:44 checking replica set: mongo-a
Tue Jan 18 16:10:44 checking replica set: mongo-b
Tue Jan 18 16:10:44 checking replica set: mongo-c
Tue Jan 18 16:11:14 [conn2] DROP DATABASE: shop
Tue Jan 18 16:11:14 [conn2] DBConfig::dropDatabase: shop
Tue Jan 18 16:11:14 [conn2] Assertion failure c s/../db/client.h 223
0x100075af2 0x100084e3e 0x10015e608 0x10015f51a 0x1001dd762 0x1000eacd3 0x1001b7058 0x1001e6ef8 0x1001f54bf 0x1000ed3a3 0x10021a314 0x7fff8695d456 0x7fff8695d309
0 mongos 0x0000000100075af2 _ZN5mongo12sayDbContextEPKc + 178
1 mongos 0x0000000100084e3e _ZN5mongo8assertedEPKcS1_j + 286
2 mongos 0x000000010015e608 _ZN5mongo12ConfigServer9logChangeERKSsS2_RKNS_7BSONObjE + 2840
3 mongos 0x000000010015f51a _ZN5mongo8DBConfig12dropDatabaseERSs + 170
4 mongos 0x00000001001dd762 _ZN5mongo15dbgrid_pub_cmds9DropDBCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb + 402
5 mongos 0x00000001000eacd3 _ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE + 1347
6 mongos 0x00000001001b7058 _ZN5mongo14SingleStrategy7queryOpERNS_7RequestE + 1192
7 mongos 0x00000001001e6ef8 _ZN5mongo7Request7processEi + 776
8 mongos 0x00000001001f54bf _ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE + 319
9 mongos 0x00000001000ed3a3 _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 419
10 mongos 0x000000010021a314 thread_proxy + 132
11 libSystem.B.dylib 0x00007fff8695d456 _pthread_start + 331
12 libSystem.B.dylib 0x00007fff8695d309 thread_start + 13
Tue Jan 18 16:11:14 [conn2] not logging config change: Gerhard-Balthasars-iMac.local-2011-01-18T15:11:14-0 assertion s/../db/client.h:223
Tue Jan 18 16:11:14 [conn2] Assertion failure c s/../db/client.h 223
0x100075af2 0x100084e3e 0x10015e608 0x100180a11 0x100159f95 0x10015faf0 0x1001dd762 0x1000eacd3 0x1001b7058 0x1001e6ef8 0x1001f54bf 0x1000ed3a3 0x10021a314 0x7fff8695d456 0x7fff8695d309
0 mongos 0x0000000100075af2 _ZN5mongo12sayDbContextEPKc + 178
1 mongos 0x0000000100084e3e _ZN5mongo8assertedEPKcS1_j + 286
2 mongos 0x000000010015e608 _ZN5mongo12ConfigServer9logChangeERKSsS2_RKNS_7BSONObjE + 2840
3 mongos 0x0000000100180a11 _ZN5mongo12ChunkManager4dropEN5boost10shared_ptrIS0_EE + 161
4 mongos 0x0000000100159f95 _ZN5mongo8DBConfig23_dropShardedCollectionsERiRSt3setINS_5ShardESt4lessIS3_ESaIS3_EERSs + 581
5 mongos 0x000000010015faf0 _ZN5mongo8DBConfig12dropDatabaseERSs + 1664
6 mongos 0x00000001001dd762 _ZN5mongo15dbgrid_pub_cmds9DropDBCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb + 402
7 mongos 0x00000001000eacd3 _ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE + 1347
8 mongos 0x00000001001b7058 _ZN5mongo14SingleStrategy7queryOpERNS_7RequestE + 1192
9 mongos 0x00000001001e6ef8 _ZN5mongo7Request7processEi + 776
10 mongos 0x00000001001f54bf _ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE + 319
11 mongos 0x00000001000ed3a3 _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 419
12 mongos 0x000000010021a314 thread_proxy + 132
13 libSystem.B.dylib 0x00007fff8695d456 _pthread_start + 331
14 libSystem.B.dylib 0x00007fff8695d309 thread_start + 13
Tue Jan 18 16:11:14 [conn2] not logging config change: Gerhard-Balthasars-iMac.local-2011-01-18T15:11:14-1 assertion s/../db/client.h:223
Tue Jan 18 16:11:14 [conn2] Assertion failure c s/../db/client.h 223
0x100075af2 0x100084e3e 0x10015e608 0x10018146e 0x100159f95 0x10015faf0 0x1001dd762 0x1000eacd3 0x1001b7058 0x1001e6ef8 0x1001f54bf 0x1000ed3a3 0x10021a314 0x7fff8695d456 0x7fff8695d309
0 mongos 0x0000000100075af2 _ZN5mongo12sayDbContextEPKc + 178
1 mongos 0x0000000100084e3e _ZN5mongo8assertedEPKcS1_j + 286
2 mongos 0x000000010015e608 _ZN5mongo12ConfigServer9logChangeERKSsS2_RKNS_7BSONObjE + 2840
3 mongos 0x000000010018146e _ZN5mongo12ChunkManager4dropEN5boost10shared_ptrIS0_EE + 2814
4 mongos 0x0000000100159f95 _ZN5mongo8DBConfig23_dropShardedCollectionsERiRSt3setINS_5ShardESt4lessIS3_ESaIS3_EERSs + 581
5 mongos 0x000000010015faf0 _ZN5mongo8DBConfig12dropDatabaseERSs + 1664
6 mongos 0x00000001001dd762 _ZN5mongo15dbgrid_pub_cmds9DropDBCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb + 402
7 mongos 0x00000001000eacd3 _ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE + 1347
8 mongos 0x00000001001b7058 _ZN5mongo14SingleStrategy7queryOpERNS_7RequestE + 1192
9 mongos 0x00000001001e6ef8 _ZN5mongo7Request7processEi + 776
10 mongos 0x00000001001f54bf _ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE + 319
11 mongos 0x00000001000ed3a3 _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 419
12 mongos 0x000000010021a314 thread_proxy + 132
13 libSystem.B.dylib 0x00007fff8695d456 _pthread_start + 331
14 libSystem.B.dylib 0x00007fff8695d309 thread_start + 13
Tue Jan 18 16:11:14 [conn2] not logging config change: Gerhard-Balthasars-iMac.local-2011-01-18T15:11:14-2 assertion s/../db/client.h:223
Tue Jan 18 16:11:14 [conn2] DBConfig::dropDatabase: shop dropped sharded collections: 1
Tue Jan 18 16:11:14 [conn2] DBConfig::dropDatabase: shop dropped sharded collections: 0
Tue Jan 18 16:11:14 [conn2] Assertion failure c s/../db/client.h 223
0x100075af2 0x100084e3e 0x10015e608 0x10015fd40 0x1001dd762 0x1000eacd3 0x1001b7058 0x1001e6ef8 0x1001f54bf 0x1000ed3a3 0x10021a314 0x7fff8695d456 0x7fff8695d309
0 mongos 0x0000000100075af2 _ZN5mongo12sayDbContextEPKc + 178
1 mongos 0x0000000100084e3e _ZN5mongo8assertedEPKcS1_j + 286
2 mongos 0x000000010015e608 _ZN5mongo12ConfigServer9logChangeERKSsS2_RKNS_7BSONObjE + 2840
3 mongos 0x000000010015fd40 _ZN5mongo8DBConfig12dropDatabaseERSs + 2256
4 mongos 0x00000001001dd762 _ZN5mongo15dbgrid_pub_cmds9DropDBCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb + 402
5 mongos 0x00000001000eacd3 _ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE + 1347
6 mongos 0x00000001001b7058 _ZN5mongo14SingleStrategy7queryOpERNS_7RequestE + 1192
7 mongos 0x00000001001e6ef8 _ZN5mongo7Request7processEi + 776
8 mongos 0x00000001001f54bf _ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE + 319
9 mongos 0x00000001000ed3a3 _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 419
10 mongos 0x000000010021a314 thread_proxy + 132
11 libSystem.B.dylib 0x00007fff8695d456 _pthread_start + 331
12 libSystem.B.dylib 0x00007fff8695d309 thread_start + 13
Tue Jan 18 16:11:14 [conn2] not logging config change: Gerhard-Balthasars-iMac.local-2011-01-18T15:11:14-3 assertion s/../db/client.h:223

Logs from mongo-int1:

1. 1. mongo-int1: Status
  2. mongo-int1: Logs
  3. mongo-int1: Process mongo-a log /home/mongodb/mongodb/logs/mongo-int1_mongo-a.log:
    Tue Jan 18 16:06:57 MongoDB starting : pid=17529 port=10000 dbpath=/home/mongodb/mongodb/data/mongo-int1_mongo-a 64-bit

- NOTE: This is a development version (1.7.4) of MongoDB.
- Not recommended for production.

Tue Jan 18 16:06:57 db version v1.7.4, pdfile version 4.5
Tue Jan 18 16:06:57 git version: fecf14def3bd51b91e0af55d61e9de2b9f918dcf
Tue Jan 18 16:06:57 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 18 16:06:58 [initandlisten] waiting for connections on port 10000
Tue Jan 18 16:06:58 [websvr] web admin interface listening on port 11000
Tue Jan 18 16:06:58 [initandlisten] connection accepted from 127.0.0.1:35171 #1
Tue Jan 18 16:06:58 [initandlisten] connection accepted from 10.0.12.180:45561 #2
Tue Jan 18 16:06:58 [startReplSets] replSet STARTUP2
Tue Jan 18 16:06:58 [rs Manager] replSet can't see a majority, will not try to elect self
Tue Jan 18 16:06:58 [ReplSetHealthPollTask] replSet info mongo-int2:10000 is up
Tue Jan 18 16:06:58 [ReplSetHealthPollTask] replSet member mongo-int2:10000 SECONDARY
Tue Jan 18 16:06:58 [ReplSetHealthPollTask] replSet info mongo-int3:10000 is up
Tue Jan 18 16:06:58 [ReplSetHealthPollTask] replSet member mongo-int3:10000 PRIMARY
Tue Jan 18 16:06:58 [rs Manager] replSet info electSelf 0
Tue Jan 18 16:06:58 [rs Manager] replSet couldn't elect self, only received -9999 votes
Tue Jan 18 16:06:58 [initandlisten] connection accepted from 10.0.12.181:17947 #3
Tue Jan 18 16:06:59 [replica set sync] building new index on

{ _id: 1 }

for local.me
Tue Jan 18 16:06:59 [replica set sync] done for 0 records 0secs
Tue Jan 18 16:06:59 [replica set sync] replSet SECONDARY
Tue Jan 18 16:07:00 [initandlisten] connection accepted from 10.0.12.182:30750 #4
Tue Jan 18 16:07:40 [initandlisten] connection accepted from 127.0.0.1:35192 #5
Tue Jan 18 16:07:51 [conn5] end connection 127.0.0.1:35192
Tue Jan 18 16:08:01 [initandlisten] connection accepted from 127.0.0.1:35194 #6
Tue Jan 18 16:08:21 [conn6] end connection 127.0.0.1:35194
Tue Jan 18 16:09:56 [initandlisten] connection accepted from 10.1.50.157:59362 #7

1. 1. mongo-int1: Process mongo-b log /home/mongodb/mongodb/logs/mongo-int1_mongo-b.log:
    Tue Jan 18 16:07:18 MongoDB starting : pid=17569 port=10001 dbpath=/home/mongodb/mongodb/data/mongo-int1_mongo-b 64-bit

- NOTE: This is a development version (1.7.4) of MongoDB.
- Not recommended for production.

Tue Jan 18 16:07:18 db version v1.7.4, pdfile version 4.5
Tue Jan 18 16:07:18 git version: fecf14def3bd51b91e0af55d61e9de2b9f918dcf
Tue Jan 18 16:07:18 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 18 16:07:18 [initandlisten] waiting for connections on port 10001
Tue Jan 18 16:07:18 [websvr] web admin interface listening on port 11001
Tue Jan 18 16:07:18 [initandlisten] connection accepted from 127.0.0.1:33274 #1
Tue Jan 18 16:07:18 [initandlisten] connection accepted from 10.0.12.180:44932 #2
Tue Jan 18 16:07:18 [startReplSets] replSet STARTUP2
Tue Jan 18 16:07:18 [rs Manager] replSet can't see a majority, will not try to elect self
Tue Jan 18 16:07:18 [ReplSetHealthPollTask] replSet info mongo-int2:10001 is up
Tue Jan 18 16:07:18 [ReplSetHealthPollTask] replSet member mongo-int2:10001 PRIMARY
Tue Jan 18 16:07:18 [ReplSetHealthPollTask] replSet info mongo-int3:10001 is up
Tue Jan 18 16:07:18 [ReplSetHealthPollTask] replSet member mongo-int3:10001 SECONDARY
Tue Jan 18 16:07:19 [replica set sync] replSet SECONDARY
Tue Jan 18 16:07:19 [initandlisten] connection accepted from 10.0.12.181:20074 #3
Tue Jan 18 16:07:20 [initandlisten] connection accepted from 10.0.12.182:33812 #4
Tue Jan 18 16:07:52 [initandlisten] connection accepted from 127.0.0.1:33289 #5
Tue Jan 18 16:07:59 [conn5] end connection 127.0.0.1:33289
Tue Jan 18 16:09:56 [initandlisten] connection accepted from 10.1.50.157:59368 #6
Tue Jan 18 16:10:46 [replica set sync] CMD: drop shop.assortment
Tue Jan 18 16:10:46 [replica set sync] dropDatabase shop
Tue Jan 18 16:10:46 [replica set sync] dropDatabase shop

1. 1. mongo-int1: Process mongo-c log /home/mongodb/mongodb/logs/mongo-int1_mongo-c.log:
    Tue Jan 18 16:07:31 MongoDB starting : pid=17605 port=10002 dbpath=/home/mongodb/mongodb/data/mongo-int1_mongo-c 64-bit

- NOTE: This is a development version (1.7.4) of MongoDB.
- Not recommended for production.

Tue Jan 18 16:07:31 db version v1.7.4, pdfile version 4.5
Tue Jan 18 16:07:31 git version: fecf14def3bd51b91e0af55d61e9de2b9f918dcf
Tue Jan 18 16:07:31 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 18 16:07:32 [initandlisten] waiting for connections on port 10002
Tue Jan 18 16:07:32 [websvr] web admin interface listening on port 11002
Tue Jan 18 16:07:32 [initandlisten] connection accepted from 127.0.0.1:50443 #1
Tue Jan 18 16:07:32 [initandlisten] connection accepted from 10.0.12.180:14672 #2
Tue Jan 18 16:07:32 [startReplSets] replSet STARTUP2
Tue Jan 18 16:07:32 [rs Manager] replSet can't see a majority, will not try to elect self
Tue Jan 18 16:07:33 [initandlisten] connection accepted from 10.0.12.182:58204 #3
Tue Jan 18 16:07:33 [initandlisten] connection accepted from 10.0.12.181:41090 #4
Tue Jan 18 16:07:34 [ReplSetHealthPollTask] replSet info mongo-int3:10002 is up
Tue Jan 18 16:07:34 [ReplSetHealthPollTask] replSet member mongo-int3:10002 PRIMARY
Tue Jan 18 16:07:34 [ReplSetHealthPollTask] replSet info mongo-int2:10002 is up
Tue Jan 18 16:07:34 [ReplSetHealthPollTask] replSet member mongo-int2:10002 SECONDARY
Tue Jan 18 16:07:34 [replica set sync] replSet SECONDARY
Tue Jan 18 16:09:56 [initandlisten] connection accepted from 10.1.50.157:59371 #5
Tue Jan 18 16:10:46 [replica set sync] CMD: drop shop.assortment
Tue Jan 18 16:10:46 [replica set sync] dropDatabase shop

1. 1. mongo-int1: Config-Server log /home/mongodb/mongodb/logs/mongo-int1_config.log:
    Tue Jan 18 16:07:32 MongoDB starting : pid=17639 port=9999 dbpath=/home/mongodb/mongodb/data/mongo-int1_config 64-bit

- NOTE: This is a development version (1.7.4) of MongoDB.
- Not recommended for production.

Tue Jan 18 16:07:32 db version v1.7.4, pdfile version 4.5
Tue Jan 18 16:07:32 git version: fecf14def3bd51b91e0af55d61e9de2b9f918dcf
Tue Jan 18 16:07:32 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 18 16:07:32 [initandlisten] diagLogging = 1
Tue Jan 18 16:07:32 [initandlisten] waiting for connections on port 9999
Tue Jan 18 16:07:32 [websvr] web admin interface listening on port 10999
Tue Jan 18 16:08:32 [DataFileSync] flushing diag log
Tue Jan 18 16:09:32 [DataFileSync] flushing diag log
Tue Jan 18 16:09:55 [initandlisten] connection accepted from 10.1.50.157:59356 #1
Tue Jan 18 16:09:55 [initandlisten] connection accepted from 10.1.50.157:59359 #2
Tue Jan 18 16:09:55 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:55 [conn2] fsync from getlasterror
Tue Jan 18 16:09:55 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:55 [conn2] fsync from getlasterror
Tue Jan 18 16:09:55 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:55 [conn2] fsync from getlasterror
Tue Jan 18 16:09:55 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:55 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [initandlisten] connection accepted from 10.1.50.157:59374 #3
Tue Jan 18 16:09:56 [initandlisten] connection accepted from 10.1.50.157:59376 #4
Tue Jan 18 16:09:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn3] fsync from getlasterror
Tue Jan 18 16:09:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn3] fsync from getlasterror
Tue Jan 18 16:09:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn3] fsync from getlasterror
Tue Jan 18 16:10:16 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:16 [conn3] fsync from getlasterror
Tue Jan 18 16:10:16 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:16 [conn3] fsync from getlasterror
Tue Jan 18 16:10:16 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:16 [conn3] fsync from getlasterror
Tue Jan 18 16:10:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:26 [conn3] fsync from getlasterror
Tue Jan 18 16:10:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:26 [conn3] fsync from getlasterror
Tue Jan 18 16:10:32 [DataFileSync] flushing diag log
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:56 [conn3] fsync from getlasterror
Tue Jan 18 16:10:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:56 [conn3] fsync from getlasterror
Tue Jan 18 16:11:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:11:26 [conn3] fsync from getlasterror
Tue Jan 18 16:11:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:11:26 [conn3] fsync from getlasterror
Tue Jan 18 16:11:32 [DataFileSync] flushing diag log
Tue Jan 18 16:11:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:11:56 [conn3] fsync from getlasterror
Tue Jan 18 16:11:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:11:56 [conn3] fsync from getlasterror
Tue Jan 18 16:12:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:12:26 [conn3] fsync from getlasterror
Tue Jan 18 16:12:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:12:26 [conn3] fsync from getlasterror
Tue Jan 18 16:12:32 [DataFileSync] flushing diag log
Tue Jan 18 16:12:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:12:56 [conn3] fsync from getlasterror
Tue Jan 18 16:12:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:12:57 [conn3] fsync from getlasterror
Tue Jan 18 16:13:27 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:13:27 [conn3] fsync from getlasterror
Tue Jan 18 16:13:27 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:13:27 [conn3] fsync from getlasterror
Tue Jan 18 16:13:32 [DataFileSync] flushing diag log
Tue Jan 18 16:13:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:13:56 [conn2] fsync from getlasterror
Tue Jan 18 16:13:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:13:56 [conn2] fsync from getlasterror
Tue Jan 18 16:13:57 [conn3] CMD fsync: sync:1 lock:0

Logs from mongo-int2:

1. 1. mongo-int2: Status
  2. mongo-int2: Logs
  3. mongo-int2: Process mongo-a log /home/mongodb/mongodb/logs/mongo-int2_mongo-a.log:
    Tue Jan 18 16:06:02 MongoDB starting : pid=16362 port=10000 dbpath=/home/mongodb/mongodb/data/mongo-int2_mongo-a 64-bit

- NOTE: This is a development version (1.7.4) of MongoDB.
- Not recommended for production.

Tue Jan 18 16:06:02 db version v1.7.4, pdfile version 4.5
Tue Jan 18 16:06:02 git version: fecf14def3bd51b91e0af55d61e9de2b9f918dcf
Tue Jan 18 16:06:02 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 18 16:06:03 [initandlisten] waiting for connections on port 10000
Tue Jan 18 16:06:03 [websvr] web admin interface listening on port 11000
Tue Jan 18 16:06:03 [initandlisten] connection accepted from 10.0.12.181:35644 #1
Tue Jan 18 16:06:03 [startReplSets] replSet STARTUP2
Tue Jan 18 16:06:03 [rs Manager] replSet can't see a majority, will not try to elect self
Tue Jan 18 16:06:03 [ReplSetHealthPollTask] replSet info mongo-int3:10000 is down (or slow to respond): socket exception
Tue Jan 18 16:06:05 [ReplSetHealthPollTask] replSet info mongo-int1:10000 is down (or slow to respond): socket exception
Tue Jan 18 16:06:29 [initandlisten] connection accepted from 10.0.12.182:49784 #2
Tue Jan 18 16:06:29 [initandlisten] connection accepted from 10.0.12.182:49785 #3
Tue Jan 18 16:06:32 [conn2] end connection 10.0.12.182:49784
Tue Jan 18 16:06:32 [conn3] query admin.$cmd ntoreturn:1 command:

{ replSetHeartbeat: "mongo-a", v: 1, pv: 1, checkEmpty: false, from: "mongo-int3:10000" }

reslen:135 2088ms
Tue Jan 18 16:06:32 [ReplSetHealthPollTask] replSet info mongo-int3:10000 is up
Tue Jan 18 16:06:32 [ReplSetHealthPollTask] replSet member mongo-int3:10000 STARTUP2
Tue Jan 18 16:06:33 [conn3] replSet RECOVERING
Tue Jan 18 16:06:33 [conn3] replSet info voting yea for 2
Tue Jan 18 16:06:33 [conn3] query admin.$cmd ntoreturn:1 command:

{ replSetElect: 1, set: "mongo-a", who: "mongo-int3:10000", whoid: 2, cfgver: 1, round: ObjectId('4d35ac7833116d5dd8a1df89') }

reslen:82 234ms
Tue Jan 18 16:06:34 [rs Manager] replSet not trying to elect self as responded yea to someone else recently
Tue Jan 18 16:06:34 [ReplSetHealthPollTask] replSet member mongo-int3:10000 PRIMARY
Tue Jan 18 16:06:36 [replica set sync] replSet SECONDARY
Tue Jan 18 16:06:58 [initandlisten] connection accepted from 10.0.12.180:43901 #4
Tue Jan 18 16:06:58 [conn4] end connection 10.0.12.180:43901
Tue Jan 18 16:06:58 [initandlisten] connection accepted from 10.0.12.180:43902 #5
Tue Jan 18 16:06:58 [conn5] mongo-int1:10000 is trying to elect itself but mongo-int3:10000 is already primary and more up-to-date
Tue Jan 18 16:06:58 [ReplSetHealthPollTask] replSet info mongo-int1:10000 is up
Tue Jan 18 16:06:58 [ReplSetHealthPollTask] replSet member mongo-int1:10000 RECOVERING
Tue Jan 18 16:07:00 [ReplSetHealthPollTask] replSet member mongo-int1:10000 SECONDARY
Tue Jan 18 16:09:56 [initandlisten] connection accepted from 10.1.50.157:59363 #6

1. 1. mongo-int2: Process mongo-b log /home/mongodb/mongodb/logs/mongo-int2_mongo-b.log:
    Tue Jan 18 16:06:13 MongoDB starting : pid=16406 port=10001 dbpath=/home/mongodb/mongodb/data/mongo-int2_mongo-b 64-bit

- NOTE: This is a development version (1.7.4) of MongoDB.
- Not recommended for production.

Tue Jan 18 16:06:13 db version v1.7.4, pdfile version 4.5
Tue Jan 18 16:06:13 git version: fecf14def3bd51b91e0af55d61e9de2b9f918dcf
Tue Jan 18 16:06:13 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 18 16:06:13 [initandlisten] waiting for connections on port 10001
Tue Jan 18 16:06:13 [websvr] web admin interface listening on port 11001
Tue Jan 18 16:06:13 [initandlisten] connection accepted from 127.0.0.1:27202 #1
Tue Jan 18 16:06:13 [initandlisten] connection accepted from 10.0.12.181:42158 #2
Tue Jan 18 16:06:13 [startReplSets] replSet STARTUP2
Tue Jan 18 16:06:13 [rs Manager] replSet can't see a majority, will not try to elect self
Tue Jan 18 16:06:13 [ReplSetHealthPollTask] replSet info mongo-int1:10001 is down (or slow to respond): socket exception
Tue Jan 18 16:06:15 [ReplSetHealthPollTask] replSet info mongo-int3:10001 is down (or slow to respond): socket exception
Tue Jan 18 16:06:40 [initandlisten] connection accepted from 10.0.12.182:62088 #3
Tue Jan 18 16:06:40 [conn3] end connection 10.0.12.182:62088
Tue Jan 18 16:06:40 [initandlisten] connection accepted from 10.0.12.182:62089 #4
Tue Jan 18 16:06:42 [ReplSetHealthPollTask] replSet info mongo-int3:10001 is up
Tue Jan 18 16:06:42 [ReplSetHealthPollTask] replSet member mongo-int3:10001 STARTUP2
Tue Jan 18 16:06:42 [rs Manager] replSet info electSelf 0
Tue Jan 18 16:06:42 [rs Manager] replSet PRIMARY
Tue Jan 18 16:06:43 [initandlisten] connection accepted from 10.0.12.182:62097 #5
Tue Jan 18 16:06:43 [conn5] query local.oplog.rs reslen:203 nscanned:2 { ts:

{ $gte: new Date(5563145390456307715) }

} nreturned:1 172ms
Tue Jan 18 16:06:44 [ReplSetHealthPollTask] replSet member mongo-int3:10001 SECONDARY
Tue Jan 18 16:06:47 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3641ms
Tue Jan 18 16:06:51 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:06:55 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:06:59 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:03 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:07 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:11 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:15 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:18 [initandlisten] connection accepted from 10.0.12.180:13769 #6
Tue Jan 18 16:07:18 [conn6] end connection 10.0.12.180:13769
Tue Jan 18 16:07:18 [initandlisten] connection accepted from 10.0.12.180:13770 #7
Tue Jan 18 16:07:19 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:19 [initandlisten] connection accepted from 10.0.12.180:13774 #8
Tue Jan 18 16:07:19 [ReplSetHealthPollTask] replSet info mongo-int1:10001 is up
Tue Jan 18 16:07:19 [ReplSetHealthPollTask] replSet member mongo-int1:10001 SECONDARY
Tue Jan 18 16:07:23 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3376ms
Tue Jan 18 16:07:23 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 4000ms
Tue Jan 18 16:07:27 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:27 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:31 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:31 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:35 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:35 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:39 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:39 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:43 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:43 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:47 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:47 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:51 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:51 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:55 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:55 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:59 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:07:59 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:03 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:03 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:07 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:07 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:11 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:11 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:15 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:15 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 4000ms
Tue Jan 18 16:08:19 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:19 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:23 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:23 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:27 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:27 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:31 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:31 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:35 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:35 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:39 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:39 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:43 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:43 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:47 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:47 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:51 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:51 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:55 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:55 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:08:59 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3995ms
Tue Jan 18 16:08:59 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:03 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:03 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3996ms
Tue Jan 18 16:09:07 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:07 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:11 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:11 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:15 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:15 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:19 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:19 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:23 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:23 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:27 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:27 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:31 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:31 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:35 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:35 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:39 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:39 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:43 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:43 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:47 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:47 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:51 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:51 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:55 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:55 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:56 [initandlisten] connection accepted from 10.1.50.157:59366 #9
Tue Jan 18 16:09:56 [initandlisten] connection accepted from 10.1.50.157:59369 #10
Tue Jan 18 16:09:59 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:09:59 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:03 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:03 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:07 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:07 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:11 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:11 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:15 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:15 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:19 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:19 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:23 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:23 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:27 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:27 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:31 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:31 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:35 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:35 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:39 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:39 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:43 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:43 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:46 [conn10] CMD: drop shop.assortment
Tue Jan 18 16:10:46 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:105 nreturned:1 3309ms
Tue Jan 18 16:10:46 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:105 nreturned:1 3309ms
Tue Jan 18 16:10:46 [conn10] dropDatabase shop
Tue Jan 18 16:10:46 [conn10] dropDatabase shop
Tue Jan 18 16:10:50 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3332ms
Tue Jan 18 16:10:50 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3295ms
Tue Jan 18 16:10:54 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:54 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:58 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:10:58 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:02 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:02 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:06 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:06 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:10 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:10 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:14 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:14 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:18 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:18 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:22 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:22 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:26 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:26 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:30 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:30 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:34 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:34 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 4000ms
Tue Jan 18 16:11:38 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:38 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:42 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:42 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:46 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:46 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:50 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:50 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:54 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:54 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:11:58 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3995ms
Tue Jan 18 16:11:58 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3995ms
Tue Jan 18 16:12:02 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:02 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:06 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:06 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:10 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:10 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:14 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:14 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:18 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:18 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:22 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:22 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:26 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:26 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:30 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:30 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:34 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:34 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:38 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:38 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:42 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:42 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:46 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:46 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:50 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:50 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:54 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:54 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:58 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:12:58 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:02 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:02 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:06 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:06 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:10 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:10 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:14 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 4000ms
Tue Jan 18 16:13:14 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 4000ms
Tue Jan 18 16:13:18 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:18 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:22 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:22 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:26 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:26 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:30 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:30 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:34 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:34 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:38 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:38 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:42 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:42 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:46 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:46 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:50 [conn8] getmore local.oplog.rs cid:1948833299886141132 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms
Tue Jan 18 16:13:50 [conn5] getmore local.oplog.rs cid:2549690613187076386 getMore: { ts:

{ $gte: new Date(5563145390456307715) }

} bytes:20 nreturned:0 3999ms

1. 1. mongo-int2: Process mongo-c log /home/mongodb/mongodb/logs/mongo-int2_mongo-c.log:
    Tue Jan 18 16:06:25 MongoDB starting : pid=16464 port=10002 dbpath=/home/mongodb/mongodb/data/mongo-int2_mongo-c 64-bit

- NOTE: This is a development version (1.7.4) of MongoDB.
- Not recommended for production.

1. 1. mongo-int2: Config-Server log /home/mongodb/mongodb/logs/mongo-int2_config.log:
    Tue Jan 18 16:06:25 MongoDB starting : pid=16503 port=9999 dbpath=/home/mongodb/mongodb/data/mongo-int2_config 64-bit

- NOTE: This is a development version (1.7.4) of MongoDB.
- Not recommended for production.

Tue Jan 18 16:06:25 db version v1.7.4, pdfile version 4.5
Tue Jan 18 16:06:25 git version: fecf14def3bd51b91e0af55d61e9de2b9f918dcf
Tue Jan 18 16:06:25 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 18 16:06:25 [initandlisten] diagLogging = 1
Tue Jan 18 16:06:25 [initandlisten] waiting for connections on port 9999
Tue Jan 18 16:06:25 [websvr] web admin interface listening on port 10999
Tue Jan 18 16:07:25 [DataFileSync] flushing diag log
Tue Jan 18 16:08:25 [DataFileSync] flushing diag log
Tue Jan 18 16:09:25 [DataFileSync] flushing diag log
Tue Jan 18 16:09:55 [initandlisten] connection accepted from 10.1.50.157:59357 #1
Tue Jan 18 16:09:55 [initandlisten] connection accepted from 10.1.50.157:59360 #2
Tue Jan 18 16:09:55 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:55 [conn2] fsync from getlasterror
Tue Jan 18 16:09:55 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:55 [conn2] fsync from getlasterror
Tue Jan 18 16:09:55 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:55 [conn2] fsync from getlasterror
Tue Jan 18 16:09:55 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn2] fsync from getlasterror
Tue Jan 18 16:09:56 [initandlisten] connection accepted from 10.1.50.157:59375 #3
Tue Jan 18 16:09:56 [initandlisten] connection accepted from 10.1.50.157:59378 #4
Tue Jan 18 16:09:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn3] fsync from getlasterror
Tue Jan 18 16:09:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn3] fsync from getlasterror
Tue Jan 18 16:09:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:09:56 [conn3] fsync from getlasterror
Tue Jan 18 16:10:16 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:16 [conn3] fsync from getlasterror
Tue Jan 18 16:10:16 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:16 [conn3] fsync from getlasterror
Tue Jan 18 16:10:16 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:16 [conn3] fsync from getlasterror
Tue Jan 18 16:10:25 [DataFileSync] flushing diag log
Tue Jan 18 16:10:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:26 [conn3] fsync from getlasterror
Tue Jan 18 16:10:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:26 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:46 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:46 [conn3] fsync from getlasterror
Tue Jan 18 16:10:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:56 [conn3] fsync from getlasterror
Tue Jan 18 16:10:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:10:56 [conn3] fsync from getlasterror
Tue Jan 18 16:11:25 [DataFileSync] flushing diag log
Tue Jan 18 16:11:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:11:26 [conn3] fsync from getlasterror
Tue Jan 18 16:11:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:11:26 [conn3] fsync from getlasterror
Tue Jan 18 16:11:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:11:56 [conn3] fsync from getlasterror
Tue Jan 18 16:11:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:11:56 [conn3] fsync from getlasterror
Tue Jan 18 16:12:25 [DataFileSync] flushing diag log
Tue Jan 18 16:12:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:12:26 [conn3] fsync from getlasterror
Tue Jan 18 16:12:26 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:12:26 [conn3] fsync from getlasterror
Tue Jan 18 16:12:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:12:56 [conn3] fsync from getlasterror
Tue Jan 18 16:12:56 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:12:57 [conn3] fsync from getlasterror
Tue Jan 18 16:13:25 [DataFileSync] flushing diag log
Tue Jan 18 16:13:27 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:13:27 [conn3] fsync from getlasterror
Tue Jan 18 16:13:27 [conn3] CMD fsync: sync:1 lock:0
Tue Jan 18 16:13:27 [conn3] fsync from getlasterror
Tue Jan 18 16:13:56 [conn2] CMD fsync: sync:1 lock:0
Tue Jan 18 16:13:56 [conn2] fsync from getlasterror
Tue Jan 18 16:13:56 [conn2] CMD fsync: sync:1 lock:0

Logs from mongo-int3:

1. 1. mongo-int3: Status
  2. mongo-int3: Logs
  3. mongo-int3: Process mongo-a log /home/mongodb/mongodb/logs/mongo-int3_mongo-a.log:
    Tue Jan 18 16:06:29 MongoDB starting : pid=16733 port=10000 dbpath=/home/mongodb/mongodb/data/mongo-int3_mongo-a 64-bit

- NOTE: This is a development version (1.7.4) of MongoDB.
- Not recommended for production.

Tue Jan 18 16:06:29 db version v1.7.4, pdfile version 4.5
Tue Jan 18 16:06:29 git version: fecf14def3bd51b91e0af55d61e9de2b9f918dcf
Tue Jan 18 16:06:29 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jan 18 16:06:29 [initandlisten] waiting for connections on port 10000
Tue Jan 18 16:06:29 [websvr] web admin interface listening on port 11000
Tue Jan 18 16:06:29 [initandlisten] connection accepted from 127.0.0.1:15453 #1
Tue Jan 18 16:06:29 [initandlisten] connection accepted from 10.0.12.182:23981 #2
Tue Jan 18 16:06:29 [startReplSets] replSet STARTUP2
Tue Jan 18 16:06:29 [rs Manager] replSet can't see a majority, will not try to elect self
Tue Jan 18 16:06:29 [ReplSetHealthPollTask] replSet info mongo-int1:10000 is down (or slow to respond): socket exception
Tue Jan 18 16:06:32 [ReplSetHealthPollTask] replSet info mongo-int2:10000 is up
Tue Jan 18 16:06:32 [ReplSetHealthPollTask] replSet member mongo-int2:10000 STARTUP2
Tue Jan 18 16:06:32 [initandlisten] connection accepted from 10.0.12.181:18275 #3
Tue Jan 18 16:06:32 [rs Manager] replSet info electSelf 2
Tue Jan 18 16:06:33 [rs Manager] replSet PRIMARY
Tue Jan 18 16:06:34 [ReplSetHealthPollTask] replSet member mongo-int2:10000 RECOVERING
Tue Jan 18 16:06:36 [initandlisten] connection accepted from 10.0.12.181:18285 #4
Tue Jan 18 16:06:36 [ReplSetHealthPollTask] replSet member mongo-int2:10000 SECONDARY
Tue Jan 18 16:06:38 [slaveTracking] building new index on

{ _id: 1 }

for local.slaves
Tue Jan 18 16:06:38 [slaveTracking] done for 0 records 0secs
Tue Jan 18 16:06:41 [conn4] getmore local.oplog.rs cid:8725246834805163579 getMore: { ts: