I'm running into assertion failures while running YCSB benchmark on sharded
MongoDB cluster (v2.0.4)

• 6 replica sets (only primaries w/o any secondary. Not idiomatic but this
  is just a test setup)
• 3 config servers
• 1 mongos router

This is what I did:

1. Enabled sharding on ycsb.usertable

> mongos> db.runCommand(

{ enablesharding: "ycsb" }

);
>

{ "ok" : 1 }

> mongos> db.runCommand( { shardcollection: "ycsb.usertable", key:

{ "_id": > 1 }

} );
>

{ "collectionsharded" : "ycsb.usertable", "ok" : 1 }

> mongos> db.printShardingStatus()
> — Sharding Status —
> sharding version:

{ "_id" : 1, "version" : 3 }

> shards:
>

{ "_id" : "shard0", "host" : "shard0/esd-repository1:50001" }

>

{ "_id" : "shard1", "host" : "shard1/esd-repository2:50001" }

>

{ "_id" : "shard2", "host" : "shard2/esd-repository3:50001" }

>

{ "_id" : "shard3", "host" : "shard3/esd-repository4:50001" }

>

{ "_id" : "shard4", "host" : "shard4/esd-repository5:50001" }

>

{ "_id" : "shard5", "host" : "shard5/esd-repository6:50001" }

> databases:
>

{ "_id" : "admin", "partitioned" : false, "primary" : "config" }

>

{ "_id" : "test", "partitioned" : false, "primary" : "shard2" }

>

{ "_id" : "ycsb", "partitioned" : true, "primary" : "shard2" }

> ycsb.usertable chunks:
> shard2 1
> { "_id" :

{ $minKey : 1 }

} -->> { "_id" :

{ > $maxKey : 1 }

} on : shard2

{ "t" : 1000, "i" : 0 }

2. Loaded 1M records into the collection (YCSB)

> ./bin/ycsb load mongodb -P workloads/workloada -s -p mongodb.url=esd-repository7:50004 -p mongodb.database=ycsb -p mongodb.writeConcern=strict -p recordcount=1000000 -threads 50

3. Tried dropping it, but failed at first.

> Wed May 2 16:12:20 uncaught exception: drop failed:

{ > "assertion" : "collection's metadata is undergoing changes. Please > try again.", > "assertionCode" : 13331, > "errmsg" : "db assertion failure", > "ok" : 0 > }

4. Finally I was able to drop it after a while.

5. Enabled sharding again on the same collection (which is dropped at this
stage)

6. Restarted YCSB loader, met numerous assertion failures
@client
> com.mongodb.MongoException: setShardVersion failed host:
> esd-repository3:50001

{ oldVersion: Timestamp 6000|1, assertio > n: "assertion s/d_state.cpp:551", errmsg: "db assertion failure", ok: 0.0 }

@mongos
> Wed May 2 16:21:30 [conn86] setShardVersion failed host:
> esd-repository3:50001

{ oldVersion: Timestamp 6000|1, assertion: "assertion > s/d_state.cpp:551", err[1612/10192] > ertion failure", ok: 0.0 }

> Wed May 2 16:21:30 [conn86] Assertion: 10429:setShardVersion failed host:
> esd-repository3:50001

{ oldVersion: Timestamp 6000|1, assertion: "assertion > s/d_state.cpp:551", err > msg: "db assertion failure", ok: 0.0 }

> 0x534e02 0x7f5105 0x7f4900 0x7f4900 0x7f4900 0x7f4900 0x7f4900 0x7f4900
> 0x5c3506 0x5c3f4e 0x5c10c0 0x5c1625 0x7679a7 0x769eba 0x770f63 0x7b501d
> 0x7c7671 0x5e8107 0x3392e0673d
> 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos [0x7f5105]
> /home/gunn/mongodb/bin/mongos [0x7f4900]
> /home/gunn/mongodb/bin/mongos [0x7f4900]
> /home/gunn/mongodb/bin/mongos [0x7f4900]
> /home/gunn/mongodb/bin/mongos [0x7f4900]
> /home/gunn/mongodb/bin/mongos [0x7f4900]
> /home/gunn/mongodb/bin/mongos [0x7f4900]
> /home/gunn/mongodb/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi+0x16)
> [0x5c3506
> ]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x16e)
> [0x5c3f4e]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2a0)
> [0x5c10c0]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0x85)
> [0x5c1625]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8Strategy6insertERKNS_5ShardEPKcRKNS_7BSONObjEib+0x67)
> [0x7679a7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo13ShardStrategy7_insertERNS_7RequestERNS_9DbMessageEN5boost10shared_ptrIKNS_12ChunkManagerEEE+0x7aa)
> [0x769eba]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo13ShardStrategy7writeOpEiRNS_7RequestE+0x153)
> [0x770f63]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0xdd) [0x7b501d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:21:30 [conn86] AssertionException while processing op type :
> 2002 to : ycsb.usertable :: caused by :: 10429 setShardVersion failed host:
> esd-repository3:50001

{ > oldVersion: Timestamp 6000|1, assertion: "assertion s/d_state.cpp:551", > errmsg: "db assertion failure", ok: 0.0 }

*
*
7. Loader finished prematurely because of the errors.
*
*
8. Tried to check the count of the collection, but another assertion

> mongos> db.usertable.count()
> Wed May 2 16:27:24 uncaught exception: count failed: {
> "assertion" : "setShardVersion failed host: esd-repository3:50001
>

{ oldVersion: Timestamp 18000|1, ns: \"ycsb.usertable\", newVersion: > Timestamp 12000|1, globalVersion: Timestamp 12000|0, errmsg: \"this > connection already had a newer version of collection 'ycsb.usertable'\", > ok: 0.0 }

",
> "assertionCode" : 10429,
> "errmsg" : "db assertion failure",
> "ok" : 0
> }

9. After some while, it worked anyway. I dropped the collection.

> mongos> db.usertable.count()
> 402581
> mongos> db.usertable.drop()
> true
> mongos> db.usertable.drop()
> false

10. stats() on the collection which should not exist by now, but I keep
seeing this.

> mongos> db.usertable.stats()
> Wed May 2 16:45:01 uncaught exception: error

{ "$err" : "too many > attempts to update config, failing", "code" : 10195 }

> mongos> db.usertable.stats()
> Wed May 2 16:45:24 uncaught exception: error

{ "$err" : "too many > attempts to update config, failing", "code" : 10195 }

something like these on mongos log
> Wed May 2 16:46:02 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(4 attempts remain)
> [127/9584]
> Wed May 2 16:46:02 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(3 attempts remain)
> Wed May 2 16:46:02 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7c7671 0x5e8107
> 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:02 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> Wed May 2 16:46:02 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(2 attempts remain)
> Wed May 2 16:46:02 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7c7671 0x5e8107
> 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:02 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> Wed May 2 16:46:02 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(1 attempts remain)
> Wed May 2 16:46:02 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7c7671 0x5e8107
> 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:02 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> [88/9584]
> Wed May 2 16:46:02 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(0 attempts remain)
> Wed May 2 16:46:02 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7c7671 0x5e8107
> 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:02 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> Wed May 2 16:46:02 [conn3] ns: EOO command failed because of stale config
> attempt: 0
> Wed May 2 16:46:02 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(4 attempts remain)
> Wed May 2 16:46:02 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(3 attempts remain)
> Wed May 2 16:46:02 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7b52ad 0x7c7671 0x5e8107
> 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:02 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> Wed May 2 16:46:02 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(2 attempts remain)
> Wed May 2 16:46:02 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7b52ad 0x7c7671 0x5e8107
> 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:02 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> [45/9584]
> Wed May 2 16:46:02 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(1 attempts remain)
> Wed May 2 16:46:02 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7b52ad 0x7c7671 0x5e8107
> 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:02 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> Wed May 2 16:46:02 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(0 attempts remain)
> Wed May 2 16:46:02 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7b52ad 0x7c7671 0x5e8107
> 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:02 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> Wed May 2 16:46:02 [conn3] ns: EOO command failed because of stale config
> attempt: 1
> Wed May 2 16:46:03 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(4 attempts remain)
> [14/9584]
> Wed May 2 16:46:03 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(3 attempts remain)
> Wed May 2 16:46:03 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7b52ad 0x7b52ad 0x7c7671
> 0x5e8107 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:03 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> Wed May 2 16:46:03 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(2 attempts remain)
> Wed May 2 16:46:03 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7b52ad 0x7b52ad 0x7c7671
> 0x5e8107 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:03 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> Wed May 2 16:46:03 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(1 attempts remain)
> Wed May 2 16:46:03 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7b52ad 0x7b52ad 0x7c7671
> 0x5e8107 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:03 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> Wed May 2 16:46:03 [conn3] retrying command:

{ collstats: "usertable", > scale: undefined }

(0 attempts remain)
> Wed May 2 16:46:03 [conn3] Assertion: 10181:not sharded:EOO
> 0x534e02 0x71303d 0x713e54 0x7789d6 0x7b50c7 0x7b52ad 0x7b52ad 0x7c7671
> 0x5e8107 0x3392e0673d 0x33926d3d1d
> /home/gunn/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112)
> [0x534e02]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsbb+0x195d)
> [0x71303d]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo8DBConfig23getChunkManagerIfExistsERKSsbb+0x34)
> [0x713e54]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x1696)
> [0x7789d6]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x187)
> [0x7b50c7]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x36d)
> [0x7b52ad]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)
> [0x7c7671]
> /home/gunn/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
> [0x5e8107]
> /lib64/libpthread.so.0 [0x3392e0673d]
> /lib64/libc.so.6(clone+0x6d) [0x33926d3d1d]
> Wed May 2 16:46:03 [conn3] warning: chunk manager not found for EOO ::
> caused by :: 10181 not sharded:EOO
> Wed May 2 16:46:03 [conn3] ns: EOO command failed because of stale config
> attempt: 2

Logs from mongos is attached.
Thanks.

Regards,
Junegunn Choi.