[SERVER-8741] mrShardedOutput.js failing on multiple platforms - ensure transferMods after commit Created: 26/Feb/13  Updated: 11/Jul/16  Resolved: 14/May/13

Status: Closed
Project: Core Server
Component/s: MapReduce, Sharding
Affects Version/s: None
Fix Version/s: 2.2.5, 2.4.4, 2.5.0

Type: Bug Priority: Major - P3
Reporter: Ian Whalen (Inactive) Assignee: Greg Studer
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File text (1).txt    
Issue Links:
Depends
depends on SERVER-8899 upgrading ShardChunkManager without w... Closed
Duplicate
duplicates SERVER-3569 Find operation does not return all do... Closed
Related
is related to SERVER-8668 jstests/sharding/mrShardedOutput.js n... Closed
is related to SERVER-8899 upgrading ShardChunkManager without w... Closed
Operating System: ALL
Participants:

 Description   

RHEL 5.7 Enterprise Build:

http://buildbot-special.10gen.com/builders/Nightly%20Linux%2064-bit%20Subscription%20RHEL%205.7/builds/293/steps/shell_3/logs/stdio

 m30999| Tue Feb 26 11:58:11.614 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Tue Feb 26 11:58:11.614 [Balancer] skipping balancing round because balancing is disabled
Count is 199999
 m30999| Tue Feb 26 11:58:17.639 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Tue Feb 26 11:58:17.639 [Balancer] skipping balancing round because balancing is disabled
Count is 199999
assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:7)
    at doassert (src/mongo/shell/assert.js:6:1)
    at Function.assert.soon (src/mongo/shell/assert.js:110:48)
    at /data/buildslaves/Linux_64bit_Subscription_RHEL_57_Nightly/mongo/jstests/sharding/mrShardedOutput.js:75:16
Tue Feb 26 11:58:19.138 JavaScript execution failed: assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds at src/mongo/shell/assert.js:L7

Issue is inserts occurring during the critical section of a migrate not being replicated to the to shard.



 Comments   
Comment by auto [ 19/May/13 ]

Author:

{u'date': u'2013-05-14T17:49:03Z', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8741 make sure we contact from-shard at least once after commit
Branch: v2.2
https://github.com/mongodb/mongo/commit/90bca9c3ad719d3070fe1c94dc3a716aff63bb3c

Comment by auto [ 19/May/13 ]

Author:

{u'date': u'2013-05-14T17:49:03Z', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8741 make sure we contact from-shard at least once after commit
Branch: v2.4
https://github.com/mongodb/mongo/commit/0762a6a2fb8fa3d130a089697da863b095f910fd

Comment by auto [ 14/May/13 ]

Author:

{u'date': u'2013-05-14T17:49:03Z', u'name': u'Greg Studer', u'email': u'greg@10gen.com'}

Message: SERVER-8741 make sure we contact from-shard at least once after commit
Branch: master
https://github.com/mongodb/mongo/commit/f9f16ce9b2d8e10e641692c6ccdba17f00d2ed19

Comment by Tad Marshall [ 12/May/13 ]

Here is an annotated log showing the problem in a test derived from mrShardedOutput.js, with enhanced diagnostics and modified logging:

// This is the client's view of when the missing document was inserted
//
 405961 DEBUG: Saving document i: 109335 a: 145.9834531415254 at ISODate("2013-05-11T20:10:09.708Z")
 
// This is the diagnostic output showing the history of the chunk that the document should have been in,
// with the time that the document was saved (in the client's view)
//
3983444 Split/Migrate/Insert history for document "109335" { "a": 145.9834531415254 }, which is supposed to be in chunk [144.7558042127639, 148.59566348604858)
3983444 on shard0001
3983445 20:09:44.765  split  [{ "$minKey" : 1 }, { "$maxKey" : 1 }) => { [{ "$minKey" : 1 }, 141.06444478966296), [141.06444478966296, { "$maxKey" : 1 }) }
3983446 20:09:44.946  split  [141.06444478966296, { "$maxKey" : 1 }) => { [141.06444478966296, 996.1213313508779), [996.1213313508779, { "$maxKey" : 1 }) }
3983447 20:09:47.106  split  [141.06444478966296, 996.1213313508779) => { [141.06444478966296, 567.1009263023734), [567.1009263023734, 996.1213313508779) }
3983448 20:09:47.370  split  [141.06444478966296, 567.1009263023734) => { [141.06444478966296, 328.63094890490174), [328.63094890490174, 567.1009263023734) }
3983449 20:09:47.928  split  [141.06444478966296, 328.63094890490174) => { [141.06444478966296, 224.41339562647045), [224.41339562647045, 328.63094890490174) }
3983450 20:09:49.316  split  [141.06444478966296, 224.41339562647045) => { [141.06444478966296, 177.88893985562027), [177.88893985562027, 224.41339562647045) }
3983451 20:09:52.505  split  [141.06444478966296, 177.88893985562027) => { [141.06444478966296, 157.24198915995657), [157.24198915995657, 177.88893985562027) }
3983452 20:10:00.278  split  [141.06444478966296, 157.24198915995657) => { [141.06444478966296, 148.59566348604858), [148.59566348604858, 157.24198915995657) }
3983453 20:10:09.640 migrate [141.06444478966296, 148.59566348604858) :: moveChunk.start :: shard0000 => shard0001
3983454 20:10:09.708  insert document "109335" { "a": 145.9834531415254 }
3983455 20:10:09.780 migrate [141.06444478966296, 148.59566348604858) :: moveChunk.to
3983456 20:10:09.782 migrate [141.06444478966296, 148.59566348604858) :: moveChunk.commit
3983457 20:10:09.783 migrate [141.06444478966296, 148.59566348604858) :: moveChunk.from
3983458 20:10:13.299  split  [141.06444478966296, 148.59566348604858) => { [141.06444478966296, 144.7558042127639), [144.7558042127639, 148.59566348604858) }
 
// The rest is the set of log lines that tell the story of how the document was lost
 
// We've reached "steady" state (but there is nothing stopping new documents from coming in)
//
 407455  m30001| Sat May 11 16:10:09.777 [conn5] runQuery called admin.$cmd { _recvChunkStatus: 1 }
 407456  m30001| Sat May 11 16:10:09.777 [conn5] run command admin.$cmd { _recvChunkStatus: 1 }
 407457  m30001| Sat May 11 16:10:09.777 [conn5] command: { _recvChunkStatus: 1 }
 407458  m30001| Sat May 11 16:10:09.777 [conn5] command admin.$cmd command: { _recvChunkStatus: 1 } ntoreturn:1 keyUpdates:0  reslen:265 0ms
 407459  m30000| Sat May 11 16:10:09.777 [conn3] insert test.foo ninserted:1 keyUpdates:0 locks(micros) w:474 0ms
 407460  m29000| Sat May 11 16:10:09.777 [conn14] runQuery called config.locks { _id: "test.foo" }
 407461  m30000| Sat May 11 16:10:09.777 [conn7] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "localhost:30000", min: { a: 141.064444
 407461 789663 }, max: { a: 148.5956634860486 }, shardKeyPattern: { a: 1.0 }, state: "steady", counts: { cloned: 807, clonedBytes: 877209, catchup: 0, steady: 0
 407461  }, ok: 1.0 } my mem used: 0
 
// progress line expanded:
//
 407461  m30000| Sat May 11 16:10:09.777 [conn7] moveChunk data transfer progress: { active: true,
                                                                                     ns: "test.foo",
                                                                                     from: "localhost:30000",
                                                                                     min: { a: 141.064444789663 },
                                                                                     max: { a: 148.5956634860486 },
                                                                                     shardKeyPattern: { a: 1.0 },
                                                                                     state: "steady",
                                                                                     counts: { cloned: 807,
                                                                                               clonedBytes: 877209,
                                                                                               catchup: 0,
                                                                                               steady: 0}, 
                                                                                     ok: 1.0 }
                                                                                    my mem used: 0
 
 407462  m30000| Sat May 11 16:10:09.777 [conn7] About to check if it is safe to enter critical section
 407463  m29000| Sat May 11 16:10:09.777 [conn14] query config.locks query: { _id: "test.foo" } ntoreturn:1 idhack:1 keyUpdates:0 locks(micros) r:19 reslen:246
 407463 0ms
 407464  m30000| Sat May 11 16:10:09.777 [conn3] logging op:{ ts: Timestamp 1368303009000|3603, op: "i", ns: "test.foo", o: { _id: ObjectId('518ea5a154436fef5a6
 407464 a00da'), a: 659.0619841590524, i: 109334.0, t: new Date(1368303009708), y: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 407464 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } }
 407465  m30000| Sat May 11 16:10:09.777 [conn3] insert test.foo ninserted:1 keyUpdates:0 locks(micros) w:184 0ms
 
// We're entering the critical section just as the "missing" document (109335) arrives
//
 407466  m30000| Sat May 11 16:10:09.777 [conn7] About to enter migrate critical section
 407467  m30001| Sat May 11 16:10:09.777 [conn3] insert test.foo ninserted:1 keyUpdates:0 locks(micros) w:413 0ms
 407468  m30000| Sat May 11 16:10:09.777 [conn3] logging op:{ ts: Timestamp 1368303009000|3604, op: "i", ns: "test.foo", o: { _id: ObjectId('518ea5a154436fef5a6
 407468 a00db'), a: 145.9834531415254, i: 109335.0, t: new Date(1368303009708), y: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 407468 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } }
 
// logOp line expanded -- this is the missing document:
//
 407468  m30000| Sat May 11 16:10:09.777 [conn3] logging op:{ ts: Timestamp 1368303009000|3604,
                                                              op: "i",
                                                              ns: "test.foo",
                                                              o: { _id: ObjectId('518ea5a154436fef5a6a00db'),
                                                                   a: 145.9834531415254,
                                                                   i: 109335.0,
                                                                   t: new Date(1368303009708),
                                                                   y: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..."
                                                                 }
                                                            }
 
// It is inserted normally ...
//
 407469  m30000| Sat May 11 16:10:09.777 [conn3] insert test.foo ninserted:1 keyUpdates:0 locks(micros) w:169 0ms
 
// ... THEN we change the version number
//
 407470  m30000| Sat May 11 16:10:09.777 [conn7] moveChunk setting version to: 14|0||518ea5881144669cbae6d5d8
 
// The following document (109336) is queued for writeback ...
//
 407471  m30000| Sat May 11 16:10:09.777 [conn3] connection sharding metadata does not match for collection test.foo, will retry (wanted : 14|0||518ea5881144669
 407471 cbae6d5d8, received : 13|1||518ea5881144669cbae6d5d8) (queuing writeback)
 407472  m30001| Sat May 11 16:10:09.777 [conn3] logging op:{ ts: Timestamp 1368303009000|1415, op: "i", ns: "test.foo", o: { _id: ObjectId('518ea5a154436fef5a6
 407472 a01c4'), a: 31.74537676386535, i: 109568.0, t: new Date(1368303009763), y: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 407472 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." } }
 407473  m30001| Sat May 11 16:10:09.777 [conn6] runQuery called admin.$cmd { _recvChunkCommit: 1 }
 407474  m30001| Sat May 11 16:10:09.777 [conn3] insert test.foo ninserted:1 keyUpdates:0 locks(micros) w:214 0ms
 407475  m30001| Sat May 11 16:10:09.777 [conn6] run command admin.$cmd { _recvChunkCommit: 1 }
 407476  m30001| Sat May 11 16:10:09.777 [conn6] command: { _recvChunkCommit: 1 }
 407477  m30001| Sat May 11 16:10:09.777 [conn6] Waiting for commit to finish
 
// ... here ...
//
 407478  m30000| Sat May 11 16:10:09.777 [conn3] writeback queued for op: insert len: 1116 ns: test.foo{ _id: ObjectId('518ea5a154436fef5a6a00dc'), a: 785.41002
 407478 49242038, i: 109336.0, t: new Date(1368303009708), y: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 407478 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa..." }
 407479  m30000| Sat May 11 16:10:09.777 [conn3] writing back msg with len: 1116 op: 2002
 407480  m30000| Sat May 11 16:10:09.778 [conn3] insert test.foo keyUpdates:0 locks(micros) w:202 0ms
 
// The chunks collection on the config server is updated here:
//
 407682  m29000| Sat May 11 16:10:09.782 [conn10] logging op:{ ts: Timestamp 1368303009000|23, op: "c", ns: "config.$cmd", o: { applyOps: [ { op: "u", b: false,
 407682  ns: "config.chunks", o: { _id: "test.foo-a_141.064444789663", lastmod: Timestamp 14000|0, lastmodEpoch: ObjectId('518ea5881144669cbae6d5d8'), ns: "test
 407682 .foo", min: { a: 141.064444789663 }, max: { a: 148.5956634860486 }, shard: "shard0001" }, o2: { _id: "test.foo-a_141.064444789663" } }, { op: "u", b: fa
 407682 lse, ns: "config.chunks", o: { _id: "test.foo-a_148.5956634860486", lastmod: Timestamp 14000|1, lastmodEpoch: ObjectId('518ea5881144669cbae6d5d8'), ns:
 407682 "test.foo", min: { a: 148.5956634860486 }, max: { a: 157.2419891599566 }, shard: "shard0000" }, o2: { _id: "test.foo-a_148.5956634860486" } } ] } }
 407683  m29000| Sat May 11 16:10:09.782 [conn10] command config.$cmd command: { applyOps: [ { op: "u", b: false, ns: "config.chunks", o: { _id: "test.foo-a_141
 407683 .064444789663", lastmod: Timestamp 14000|0, lastmodEpoch: ObjectId('518ea5881144669cbae6d5d8'), ns: "test.foo", min: { a: 141.064444789663 }, max: { a:
 407683 148.5956634860486 }, shard: "shard0001" }, o2: { _id: "test.foo-a_141.064444789663" } }, { op: "u", b: false, ns: "config.chunks", o: { _id: "test.foo-a
 407683 _148.5956634860486", lastmod: Timestamp 14000|1, lastmodEpoch: ObjectId('518ea5881144669cbae6d5d8'), ns: "test.foo", min: { a: 148.5956634860486 }, max:
 407683  { a: 157.2419891599566 }, shard: "shard0000" }, o2: { _id: "test.foo-a_148.5956634860486" } } ], preCondition: [ { ns: "config.chunks", q: { query: { n
 407683 s: "test.foo" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 13000|11 } } ] } ntoreturn:1 keyUpdates:0 locks(micros) W:778 reslen:72 0ms
 
// expanded:
 407682  m29000| Sat May 11 16:10:09.782 [conn10] logging op:{ ts: Timestamp 1368303009000|23,
                                                               op: "c",
                                                               ns: "config.$cmd",
                                                               o: { applyOps: [
                                                                        { op: "u",
                                                                          b: false,
                                                                          ns: "config.chunks",
                                                                          o: { _id: "test.foo-a_141.064444789663",
                                                                               lastmod: Timestamp 14000|0,
                                                                               lastmodEpoch: ObjectId('518ea5881144669cbae6d5d8'),
                                                                               ns: "test.foo",
                                                                               min: { a: 141.064444789663 },
                                                                               max: { a: 148.5956634860486 },
                                                                               shard: "shard0001" },
                                                                          o2: { _id: "test.foo-a_141.064444789663" }
                                                                        },
                                                                        { op: "u",
                                                                          b: false,
                                                                          ns: "config.chunks",
                                                                          o: { _id: "test.foo-a_148.5956634860486",
                                                                               lastmod: Timestamp 14000|1,
                                                                               lastmodEpoch: ObjectId('518ea5881144669cbae6d5d8'),
                                                                               ns:"test.foo",
                                                                               min: { a: 148.5956634860486 },
                                                                               max: { a: 157.2419891599566 },
                                                                               shard: "shard0000" },
                                                                          o2: { _id: "test.foo-a_148.5956634860486" }
                                                                        }
                                                                    ]
                                                               }
                                                             }
 407683  m29000| Sat May 11 16:10:09.782 [conn10] command config.$cmd command: { applyOps: [
                                                                                    { op: "u",
                                                                                      b: false,
                                                                                      ns: "config.chunks",
                                                                                      o: { _id: "test.foo-a_141.064444789663",
                                                                                           lastmod: Timestamp 14000|0,
                                                                                           lastmodEpoch: ObjectId('518ea5881144669cbae6d5d8'),
                                                                                           ns: "test.foo",
                                                                                           min: { a: 141.064444789663 },
                                                                                           max: { a:148.5956634860486 },
                                                                                           shard: "shard0001" },
                                                                                      o2: { _id: "test.foo-a_141.064444789663" }
                                                                                    },
                                                                                    { op: "u",
                                                                                      b: false,
                                                                                      ns: "config.chunks",
                                                                                      o: { _id: "test.foo-a_148.5956634860486",
                                                                                           lastmod: Timestamp 14000|1,
                                                                                           lastmodEpoch: ObjectId('518ea5881144669cbae6d5d8'),
                                                                                           ns: "test.foo",
                                                                                           min: { a: 148.5956634860486 },
                                                                                           max: { a: 157.2419891599566 },
                                                                                           shard: "shard0000" },
                                                                                      o2: { _id: "test.foo-a_148.5956634860486" }
                                                                                    }
                                                                                 ],
                                                                                 preCondition: [
                                                                                    { ns: "config.chunks",
                                                                                      q: { query: { ns: "test.foo" },
                                                                                           orderby: { lastmod: -1 }
                                                                                         },
                                                                                      res: { lastmod: Timestamp 13000|11 }
                                                                                    }
                                                                                 ]
                                                                               } ntoreturn:1 keyUpdates:0 locks(micros) W:778 reslen:72 0ms
 
// The missing document is never migrated ...
 
// ... but later, it is deleted ...
//
 415241  m30000| Sat May 11 16:10:10.049 [cleanupOldData-518ea5a1fe658ac977ccbf15] logging op:{ ts: Timestamp 1368303010000|334, op: "d", ns: "test.foo", fromMi
 415241 grate: true, o: { _id: ObjectId('518ea5a154436fef5a6a00db') } }

Comment by Tad Marshall [ 30/Apr/13 ]

Linux 32-bit again:
http://buildbot.mongodb.org/builders/Nightly%20Linux%2032-bit/builds/1309/steps/test_11/logs/stdio
http://buildlogs.mongodb.org/Nightly%20Linux%2032-bit/builds/1309/test/sharding/mrShardedOutput.js

Setup FAILURE: testDB.foo.find().itcount() = 59997 after saving 60000 documents -- (will assert after diagnostics)
// ... snip ...
*** ====> mongos could not find document 32781 with shard key 143.11787602491677: not found on either shard
*** ====> mongos could not find document 32869 with shard key 151.7694746144116: not found on either shard
*** ====> mongos could not find document 32872 with shard key 145.90478944592178: not found on either shard
// ... snip ...
========> Finished checking 60000
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"version" : 3,
	"minCompatibleVersion" : 3,
	"currentVersion" : 4,
	"clusterId" : ObjectId("51800da08d146f1dfa09bf6b")
}
  shards:
	{  "_id" : "shard0000",  "host" : "localhost:30000" }
	{  "_id" : "shard0001",  "host" : "localhost:30001" }
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard0001" }
		test.foo
			shard key: { "a" : 1 }
			chunks:
				shard0000	20
				shard0001	65
			{ "a" : { "$minKey" : 1 } } -->> { "a" : 0.7614018395543098 } on : shard0000 Timestamp(2, 0) 
			{ "a" : 0.7614018395543098 } -->> { "a" : 12.480927631258965 } on : shard0000 Timestamp(10, 12) 
			{ "a" : 12.480927631258965 } -->> { "a" : 28.130482183769345 } on : shard0000 Timestamp(10, 13) 
			{ "a" : 28.130482183769345 } -->> { "a" : 36.376196425408125 } on : shard0000 Timestamp(12, 26) 
			{ "a" : 36.376196425408125 } -->> { "a" : 44.699640944600105 } on : shard0000 Timestamp(12, 27) 
			{ "a" : 44.699640944600105 } -->> { "a" : 54.042244562879205 } on : shard0000 Timestamp(12, 24) 
			{ "a" : 54.042244562879205 } -->> { "a" : 69.17390692979097 } on : shard0000 Timestamp(12, 25) 
			{ "a" : 69.17390692979097 } -->> { "a" : 78.95258325152099 } on : shard0000 Timestamp(12, 20) 
			{ "a" : 78.95258325152099 } -->> { "a" : 89.7432656493038 } on : shard0000 Timestamp(12, 21) 
			{ "a" : 89.7432656493038 } -->> { "a" : 100.83158873021603 } on : shard0000 Timestamp(10, 26) 
			{ "a" : 100.83158873021603 } -->> { "a" : 115.95065728761256 } on : shard0000 Timestamp(10, 27) 
			{ "a" : 115.95065728761256 } -->> { "a" : 124.89698803983629 } on : shard0000 Timestamp(12, 22) 
			{ "a" : 124.89698803983629 } -->> { "a" : 136.38805435039103 } on : shard0000 Timestamp(12, 23) 
			{ "a" : 136.38805435039103 } -->> { "a" : 148.94994045607746 } on : shard0000 Timestamp(10, 2) 
			{ "a" : 148.94994045607746 } -->> { "a" : 164.64181942865252 } on : shard0000 Timestamp(10, 3) 
			{ "a" : 164.64181942865252 } -->> { "a" : 180.96433230675757 } on : shard0000 Timestamp(10, 0) 
			{ "a" : 180.96433230675757 } -->> { "a" : 191.67375238612294 } on : shard0000 Timestamp(11, 2) 
			{ "a" : 191.67375238612294 } -->> { "a" : 202.05870340578258 } on : shard0000 Timestamp(11, 3) 
			{ "a" : 202.05870340578258 } -->> { "a" : 211.1177744809538 } on : shard0000 Timestamp(12, 18) 
			{ "a" : 211.1177744809538 } -->> { "a" : 222.4627686664462 } on : shard0000 Timestamp(12, 19) 
			{ "a" : 222.4627686664462 } -->> { "a" : 236.36041744612157 } on : shard0001 Timestamp(12, 1) 
			{ "a" : 236.36041744612157 } -->> { "a" : 253.79521399736404 } on : shard0001 Timestamp(9, 3) 
			{ "a" : 253.79521399736404 } -->> { "a" : 263.0670985672623 } on : shard0001 Timestamp(11, 8) 
			{ "a" : 263.0670985672623 } -->> { "a" : 275.9274458512664 } on : shard0001 Timestamp(11, 9) 
			{ "a" : 275.9274458512664 } -->> { "a" : 289.8150156252086 } on : shard0001 Timestamp(9, 14) 
			{ "a" : 289.8150156252086 } -->> { "a" : 305.72284176014364 } on : shard0001 Timestamp(9, 15) 
			{ "a" : 305.72284176014364 } -->> { "a" : 318.1853659916669 } on : shard0001 Timestamp(10, 8) 
			{ "a" : 318.1853659916669 } -->> { "a" : 333.14331248402596 } on : shard0001 Timestamp(10, 9) 
			{ "a" : 333.14331248402596 } -->> { "a" : 348.8472681492567 } on : shard0001 Timestamp(1, 83) 
			{ "a" : 348.8472681492567 } -->> { "a" : 358.736558817327 } on : shard0001 Timestamp(11, 6) 
			{ "a" : 358.736558817327 } -->> { "a" : 369.64876018464565 } on : shard0001 Timestamp(11, 7) 
			{ "a" : 369.64876018464565 } -->> { "a" : 378.78237571567297 } on : shard0001 Timestamp(12, 4) 
			{ "a" : 378.78237571567297 } -->> { "a" : 393.3888382744044 } on : shard0001 Timestamp(12, 5) 
			{ "a" : 393.3888382744044 } -->> { "a" : 406.0760682914406 } on : shard0001 Timestamp(9, 6) 
			{ "a" : 406.0760682914406 } -->> { "a" : 413.77965174615383 } on : shard0001 Timestamp(12, 30) 
			{ "a" : 413.77965174615383 } -->> { "a" : 424.4644397404045 } on : shard0001 Timestamp(12, 31) 
			{ "a" : 424.4644397404045 } -->> { "a" : 436.62508577108383 } on : shard0001 Timestamp(10, 28) 
			{ "a" : 436.62508577108383 } -->> { "a" : 445.0965179130435 } on : shard0001 Timestamp(12, 28) 
			{ "a" : 445.0965179130435 } -->> { "a" : 454.32045427151024 } on : shard0001 Timestamp(12, 29) 
			{ "a" : 454.32045427151024 } -->> { "a" : 461.94622735492885 } on : shard0001 Timestamp(12, 32) 
			{ "a" : 461.94622735492885 } -->> { "a" : 472.42358746007085 } on : shard0001 Timestamp(12, 33) 
			{ "a" : 472.42358746007085 } -->> { "a" : 484.06825703568757 } on : shard0001 Timestamp(10, 14) 
			{ "a" : 484.06825703568757 } -->> { "a" : 499.0712597500533 } on : shard0001 Timestamp(10, 15) 
			{ "a" : 499.0712597500533 } -->> { "a" : 511.3720402587205 } on : shard0001 Timestamp(10, 4) 
			{ "a" : 511.3720402587205 } -->> { "a" : 520.5155103467405 } on : shard0001 Timestamp(12, 10) 
			{ "a" : 520.5155103467405 } -->> { "a" : 534.1696301475167 } on : shard0001 Timestamp(12, 11) 
			{ "a" : 534.1696301475167 } -->> { "a" : 543.5029941145331 } on : shard0001 Timestamp(12, 12) 
			{ "a" : 543.5029941145331 } -->> { "a" : 555.1349855959415 } on : shard0001 Timestamp(12, 13) 
			{ "a" : 555.1349855959415 } -->> { "a" : 568.5213170945644 } on : shard0001 Timestamp(9, 8) 
			{ "a" : 568.5213170945644 } -->> { "a" : 584.1243544127792 } on : shard0001 Timestamp(9, 9) 
			{ "a" : 584.1243544127792 } -->> { "a" : 593.8224827405065 } on : shard0001 Timestamp(12, 2) 
			{ "a" : 593.8224827405065 } -->> { "a" : 604.8288189340383 } on : shard0001 Timestamp(12, 3) 
			{ "a" : 604.8288189340383 } -->> { "a" : 617.1443914063275 } on : shard0001 Timestamp(10, 24) 
			{ "a" : 617.1443914063275 } -->> { "a" : 632.8016347251832 } on : shard0001 Timestamp(10, 25) 
			{ "a" : 632.8016347251832 } -->> { "a" : 642.6627153996378 } on : shard0001 Timestamp(12, 8) 
			{ "a" : 642.6627153996378 } -->> { "a" : 655.8673069812357 } on : shard0001 Timestamp(12, 9) 
			{ "a" : 655.8673069812357 } -->> { "a" : 667.5234434660524 } on : shard0001 Timestamp(10, 6) 
			{ "a" : 667.5234434660524 } -->> { "a" : 682.3369811754674 } on : shard0001 Timestamp(10, 7) 
			{ "a" : 682.3369811754674 } -->> { "a" : 694.6301823481917 } on : shard0001 Timestamp(10, 16) 
			{ "a" : 694.6301823481917 } -->> { "a" : 708.6858213879168 } on : shard0001 Timestamp(10, 17) 
			{ "a" : 708.6858213879168 } -->> { "a" : 722.4257290363312 } on : shard0001 Timestamp(9, 12) 
			{ "a" : 722.4257290363312 } -->> { "a" : 731.3909707590938 } on : shard0001 Timestamp(12, 14) 
			{ "a" : 731.3909707590938 } -->> { "a" : 740.9236812964082 } on : shard0001 Timestamp(12, 15) 
			{ "a" : 740.9236812964082 } -->> { "a" : 755.0070988945663 } on : shard0001 Timestamp(9, 4) 
			{ "a" : 755.0070988945663 } -->> { "a" : 764.6509746555239 } on : shard0001 Timestamp(12, 6) 
			{ "a" : 764.6509746555239 } -->> { "a" : 775.3659824375063 } on : shard0001 Timestamp(12, 7) 
			{ "a" : 775.3659824375063 } -->> { "a" : 785.6708085164428 } on : shard0001 Timestamp(12, 16) 
			{ "a" : 785.6708085164428 } -->> { "a" : 796.1352756246924 } on : shard0001 Timestamp(12, 17) 
			{ "a" : 796.1352756246924 } -->> { "a" : 808.3587645087391 } on : shard0001 Timestamp(10, 22) 
			{ "a" : 808.3587645087391 } -->> { "a" : 822.0081983599812 } on : shard0001 Timestamp(10, 23) 
			{ "a" : 822.0081983599812 } -->> { "a" : 835.4945846367627 } on : shard0001 Timestamp(9, 10) 
			{ "a" : 835.4945846367627 } -->> { "a" : 845.7749916706234 } on : shard0001 Timestamp(11, 4) 
			{ "a" : 845.7749916706234 } -->> { "a" : 856.7083778325468 } on : shard0001 Timestamp(11, 5) 
			{ "a" : 856.7083778325468 } -->> { "a" : 874.9041352421045 } on : shard0001 Timestamp(1, 75) 
			{ "a" : 874.9041352421045 } -->> { "a" : 885.7789670582861 } on : shard0001 Timestamp(10, 18) 
			{ "a" : 885.7789670582861 } -->> { "a" : 898.9005489274859 } on : shard0001 Timestamp(10, 19) 
			{ "a" : 898.9005489274859 } -->> { "a" : 917.2087986953557 } on : shard0001 Timestamp(1, 71) 
			{ "a" : 917.2087986953557 } -->> { "a" : 927.8832757845521 } on : shard0001 Timestamp(10, 20) 
			{ "a" : 927.8832757845521 } -->> { "a" : 940.286184893921 } on : shard0001 Timestamp(10, 21) 
			{ "a" : 940.286184893921 } -->> { "a" : 952.3370799142867 } on : shard0001 Timestamp(10, 10) 
			{ "a" : 952.3370799142867 } -->> { "a" : 965.740226674825 } on : shard0001 Timestamp(10, 11) 
			{ "a" : 965.740226674825 } -->> { "a" : 981.5583629533648 } on : shard0001 Timestamp(1, 79) 
			{ "a" : 981.5583629533648 } -->> { "a" : 988.6023679282516 } on : shard0001 Timestamp(12, 34) 
			{ "a" : 988.6023679282516 } -->> { "a" : 998.8424617331475 } on : shard0001 Timestamp(12, 35) 
			{ "a" : 998.8424617331475 } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 4) 
// ... snip ...
Count is 59997
assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:15)
    at doassert (src/mongo/shell/assert.js:6:5)
    at Function.assert.soon (src/mongo/shell/assert.js:110:60)
    at /mnt/slaves/Linux_32bit_Nightly/mongo/jstests/sharding/mrShardedOutput.js:91:16
Tue Apr 30 14:31:33.722 JavaScript execution failed: assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds at src/mongo/shell/assert.js:L7
failed to load: /mnt/slaves/Linux_32bit_Nightly/mongo/jstests/sharding/mrShardedOutput.js

Comment by Tad Marshall [ 25/Apr/13 ]

On RHEL Linux 32-bit:
http://buildbot.mongodb.org/builders/Nightly%20Linux%20RHEL%2032-bit/builds/472/steps/test_11/logs/stdio
http://buildlogs.mongodb.org/Nightly%20Linux%20RHEL%2032-bit/builds/472/test/sharding/mrShardedOutput.js

Setup FAILURE: testDB.foo.find().itcount() = 59999 after saving 60000 documents -- (will assert after diagnostics)
// ... snip ...
*** ====> mongos could not find document 41699 with shard key 226.8135747872293: not found on either shard
// ... snip ...
========> Finished checking 60000
 
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"version" : 3,
	"minCompatibleVersion" : 3,
	"currentVersion" : 4,
	"clusterId" : ObjectId("51791e3b010e407ab39d6407")
}
  shards:
	{  "_id" : "shard0000",  "host" : "localhost:30000" }
	{  "_id" : "shard0001",  "host" : "localhost:30001" }
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard0001" }
		test.foo
			shard key: { "a" : 1 }
			chunks:
				shard0000	22
				shard0001	55
			{ "a" : { "$minKey" : 1 } } -->> { "a" : 0.24301582016050816 } on : shard0000 Timestamp(2, 0) 
			{ "a" : 0.24301582016050816 } -->> { "a" : 14.294489985331893 } on : shard0000 Timestamp(9, 14) 
			{ "a" : 14.294489985331893 } -->> { "a" : 33.70938613079488 } on : shard0000 Timestamp(9, 15) 
			{ "a" : 33.70938613079488 } -->> { "a" : 44.966764748096466 } on : shard0000 Timestamp(11, 10) 
			{ "a" : 44.966764748096466 } -->> { "a" : 59.26825013011694 } on : shard0000 Timestamp(11, 11) 
			{ "a" : 59.26825013011694 } -->> { "a" : 68.83858470246196 } on : shard0000 Timestamp(12, 20) 
			{ "a" : 68.83858470246196 } -->> { "a" : 84.51949222944677 } on : shard0000 Timestamp(12, 21) 
			{ "a" : 84.51949222944677 } -->> { "a" : 93.4156677685678 } on : shard0000 Timestamp(13, 4) 
			{ "a" : 93.4156677685678 } -->> { "a" : 104.78522162884474 } on : shard0000 Timestamp(13, 5) 
			{ "a" : 104.78522162884474 } -->> { "a" : 118.8176937866956 } on : shard0000 Timestamp(9, 10) 
			{ "a" : 118.8176937866956 } -->> { "a" : 133.13065166585147 } on : shard0000 Timestamp(9, 11) 
			{ "a" : 133.13065166585147 } -->> { "a" : 146.7801067046821 } on : shard0000 Timestamp(9, 4) 
			{ "a" : 146.7801067046821 } -->> { "a" : 164.996184874326 } on : shard0000 Timestamp(9, 5) 
			{ "a" : 164.996184874326 } -->> { "a" : 181.5820604097098 } on : shard0000 Timestamp(9, 0) 
			{ "a" : 181.5820604097098 } -->> { "a" : 200.2613227814436 } on : shard0000 Timestamp(10, 0) 
			{ "a" : 200.2613227814436 } -->> { "a" : 213.1073495838791 } on : shard0000 Timestamp(11, 0) 
			{ "a" : 213.1073495838791 } -->> { "a" : 223.83665619418025 } on : shard0000 Timestamp(12, 14) 
			{ "a" : 223.83665619418025 } -->> { "a" : 235.8140607830137 } on : shard0000 Timestamp(12, 15) 
			{ "a" : 235.8140607830137 } -->> { "a" : 246.5141210705042 } on : shard0000 Timestamp(13, 0) 
			{ "a" : 246.5141210705042 } -->> { "a" : 258.59659910202026 } on : shard0000 Timestamp(14, 0) 
			{ "a" : 258.59659910202026 } -->> { "a" : 269.3021157756448 } on : shard0000 Timestamp(15, 0) 
			{ "a" : 269.3021157756448 } -->> { "a" : 283.1406856421381 } on : shard0000 Timestamp(16, 0) 
			{ "a" : 283.1406856421381 } -->> { "a" : 293.69053011760116 } on : shard0001 Timestamp(16, 1) 
			{ "a" : 293.69053011760116 } -->> { "a" : 306.5501772798598 } on : shard0001 Timestamp(12, 9) 
			{ "a" : 306.5501772798598 } -->> { "a" : 323.19938600994647 } on : shard0001 Timestamp(1, 79) 
			{ "a" : 323.19938600994647 } -->> { "a" : 339.72371579147875 } on : shard0001 Timestamp(1, 80) 
			{ "a" : 339.72371579147875 } -->> { "a" : 351.15210455842316 } on : shard0001 Timestamp(11, 4) 
			{ "a" : 351.15210455842316 } -->> { "a" : 366.4183511864394 } on : shard0001 Timestamp(11, 5) 
			{ "a" : 366.4183511864394 } -->> { "a" : 380.5887040216476 } on : shard0001 Timestamp(9, 2) 
			{ "a" : 380.5887040216476 } -->> { "a" : 397.10158156231046 } on : shard0001 Timestamp(9, 3) 
			{ "a" : 397.10158156231046 } -->> { "a" : 411.69070336036384 } on : shard0001 Timestamp(9, 20) 
			{ "a" : 411.69070336036384 } -->> { "a" : 427.055697189644 } on : shard0001 Timestamp(9, 21) 
			{ "a" : 427.055697189644 } -->> { "a" : 436.7909734137356 } on : shard0001 Timestamp(12, 18) 
			{ "a" : 436.7909734137356 } -->> { "a" : 448.46950750797987 } on : shard0001 Timestamp(12, 19) 
			{ "a" : 448.46950750797987 } -->> { "a" : 459.00204312056303 } on : shard0001 Timestamp(12, 4) 
			{ "a" : 459.00204312056303 } -->> { "a" : 471.69400099664927 } on : shard0001 Timestamp(12, 5) 
			{ "a" : 471.69400099664927 } -->> { "a" : 489.68531703576446 } on : shard0001 Timestamp(1, 71) 
			{ "a" : 489.68531703576446 } -->> { "a" : 501.6803725156933 } on : shard0001 Timestamp(11, 6) 
			{ "a" : 501.6803725156933 } -->> { "a" : 515.9879680722952 } on : shard0001 Timestamp(11, 7) 
			{ "a" : 515.9879680722952 } -->> { "a" : 530.0977067090571 } on : shard0001 Timestamp(9, 22) 
			{ "a" : 530.0977067090571 } -->> { "a" : 545.0144049245864 } on : shard0001 Timestamp(9, 23) 
			{ "a" : 545.0144049245864 } -->> { "a" : 560.3021220304072 } on : shard0001 Timestamp(9, 16) 
			{ "a" : 560.3021220304072 } -->> { "a" : 576.8860462121665 } on : shard0001 Timestamp(9, 17) 
			{ "a" : 576.8860462121665 } -->> { "a" : 591.3134992588311 } on : shard0001 Timestamp(9, 18) 
			{ "a" : 591.3134992588311 } -->> { "a" : 598.7987706903368 } on : shard0001 Timestamp(13, 8) 
			{ "a" : 598.7987706903368 } -->> { "a" : 610.3188791312277 } on : shard0001 Timestamp(13, 9) 
			{ "a" : 610.3188791312277 } -->> { "a" : 621.2622553575784 } on : shard0001 Timestamp(12, 2) 
			{ "a" : 621.2622553575784 } -->> { "a" : 632.9791105818003 } on : shard0001 Timestamp(12, 3) 
			{ "a" : 632.9791105818003 } -->> { "a" : 640.789397759363 } on : shard0001 Timestamp(13, 12) 
			{ "a" : 640.789397759363 } -->> { "a" : 658.2449551206082 } on : shard0001 Timestamp(13, 13) 
			{ "a" : 658.2449551206082 } -->> { "a" : 668.9622132107615 } on : shard0001 Timestamp(12, 10) 
			{ "a" : 668.9622132107615 } -->> { "a" : 682.3329024482518 } on : shard0001 Timestamp(12, 11) 
			{ "a" : 682.3329024482518 } -->> { "a" : 697.8907184675336 } on : shard0001 Timestamp(1, 81) 
			{ "a" : 697.8907184675336 } -->> { "a" : 714.8263640701771 } on : shard0001 Timestamp(1, 82) 
			{ "a" : 714.8263640701771 } -->> { "a" : 729.1622986085713 } on : shard0001 Timestamp(9, 8) 
			{ "a" : 729.1622986085713 } -->> { "a" : 744.0494273323566 } on : shard0001 Timestamp(9, 9) 
			{ "a" : 744.0494273323566 } -->> { "a" : 758.2104648463428 } on : shard0001 Timestamp(9, 6) 
			{ "a" : 758.2104648463428 } -->> { "a" : 766.6432915721089 } on : shard0001 Timestamp(13, 6) 
			{ "a" : 766.6432915721089 } -->> { "a" : 777.8738525230438 } on : shard0001 Timestamp(13, 7) 
			{ "a" : 777.8738525230438 } -->> { "a" : 793.2017624843866 } on : shard0001 Timestamp(1, 73) 
			{ "a" : 793.2017624843866 } -->> { "a" : 803.7758248392493 } on : shard0001 Timestamp(12, 6) 
			{ "a" : 803.7758248392493 } -->> { "a" : 817.47179152444 } on : shard0001 Timestamp(12, 7) 
			{ "a" : 817.47179152444 } -->> { "a" : 829.6936959959567 } on : shard0001 Timestamp(11, 2) 
			{ "a" : 829.6936959959567 } -->> { "a" : 844.4085505325347 } on : shard0001 Timestamp(11, 3) 
			{ "a" : 844.4085505325347 } -->> { "a" : 858.4102613385767 } on : shard0001 Timestamp(9, 12) 
			{ "a" : 858.4102613385767 } -->> { "a" : 867.6875489763916 } on : shard0001 Timestamp(13, 10) 
			{ "a" : 867.6875489763916 } -->> { "a" : 878.1193899922073 } on : shard0001 Timestamp(13, 11) 
			{ "a" : 878.1193899922073 } -->> { "a" : 888.0166695453227 } on : shard0001 Timestamp(12, 12) 
			{ "a" : 888.0166695453227 } -->> { "a" : 898.5631316900253 } on : shard0001 Timestamp(12, 13) 
			{ "a" : 898.5631316900253 } -->> { "a" : 912.4502011109143 } on : shard0001 Timestamp(9, 24) 
			{ "a" : 912.4502011109143 } -->> { "a" : 930.4029257036746 } on : shard0001 Timestamp(9, 25) 
			{ "a" : 930.4029257036746 } -->> { "a" : 944.0549090504646 } on : shard0001 Timestamp(9, 26) 
			{ "a" : 944.0549090504646 } -->> { "a" : 964.712840039283 } on : shard0001 Timestamp(9, 27) 
			{ "a" : 964.712840039283 } -->> { "a" : 980.9439948294312 } on : shard0001 Timestamp(1, 75) 
			{ "a" : 980.9439948294312 } -->> { "a" : 990.1225513312966 } on : shard0001 Timestamp(13, 2) 
			{ "a" : 990.1225513312966 } -->> { "a" : 999.5905663818121 } on : shard0001 Timestamp(13, 3) 
			{ "a" : 999.5905663818121 } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 4) 
// ... snip ...
Count is 59999
assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:15)
    at doassert (src/mongo/shell/assert.js:6:5)
    at Function.assert.soon (src/mongo/shell/assert.js:110:60)
    at /data/buildslaves/Linux_RHEL_32bit_Nightly/mongo/jstests/sharding/mrShardedOutput.js:91:16
Thu Apr 25 08:16:26.338 JavaScript execution failed: assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds at src/mongo/shell/assert.js:L7
failed to load: /data/buildslaves/Linux_RHEL_32bit_Nightly/mongo/jstests/sharding/mrShardedOutput.js

Comment by Tad Marshall [ 24/Apr/13 ]

On Linux 32-bit:
http://buildbot.mongodb.org/builders/Linux%2032-bit/builds/5655/steps/test_11/logs/stdio
http://buildlogs.mongodb.org/Linux%2032-bit/builds/5655/test/sharding/mrShardedOutput.js

Setup FAILURE: testDB.foo.find().itcount() = 29999 after saving 30000 documents -- (will assert after diagnostics)
// ... snip ...
*** ====> mongos could not find document 27663 with shard key 1.820039702579379: not found on either shard
// ... snip ...
========> Finished checking 30000
 
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"version" : 3,
	"minCompatibleVersion" : 3,
	"currentVersion" : 4,
	"clusterId" : ObjectId("5178490306bea5a755e048a0")
}
  shards:
	{  "_id" : "shard0000",  "host" : "localhost:30000" }
	{  "_id" : "shard0001",  "host" : "localhost:30001" }
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard0001" }
		test.foo
			shard key: { "a" : 1 }
			chunks:
				shard0000	1
				shard0001	37
			{ "a" : { "$minKey" : 1 } } -->> { "a" : 4.386908374726772 } on : shard0000 Timestamp(2, 0) 
			{ "a" : 4.386908374726772 } -->> { "a" : 22.8157180827111 } on : shard0001 Timestamp(2, 1) 
			{ "a" : 22.8157180827111 } -->> { "a" : 47.5725217256695 } on : shard0001 Timestamp(1, 72) 
			{ "a" : 47.5725217256695 } -->> { "a" : 70.08078531362116 } on : shard0001 Timestamp(1, 63) 
			{ "a" : 70.08078531362116 } -->> { "a" : 100.1467325259 } on : shard0001 Timestamp(1, 64) 
			{ "a" : 100.1467325259 } -->> { "a" : 122.79061460867524 } on : shard0001 Timestamp(1, 61) 
			{ "a" : 122.79061460867524 } -->> { "a" : 149.6480864007026 } on : shard0001 Timestamp(1, 62) 
			{ "a" : 149.6480864007026 } -->> { "a" : 182.19448276795447 } on : shard0001 Timestamp(1, 41) 
			{ "a" : 182.19448276795447 } -->> { "a" : 222.35085908323526 } on : shard0001 Timestamp(1, 42) 
			{ "a" : 222.35085908323526 } -->> { "a" : 247.3482855129987 } on : shard0001 Timestamp(1, 57) 
			{ "a" : 247.3482855129987 } -->> { "a" : 274.6123324614018 } on : shard0001 Timestamp(1, 58) 
			{ "a" : 274.6123324614018 } -->> { "a" : 300.37696240469813 } on : shard0001 Timestamp(1, 53) 
			{ "a" : 300.37696240469813 } -->> { "a" : 337.094459682703 } on : shard0001 Timestamp(1, 54) 
			{ "a" : 337.094459682703 } -->> { "a" : 369.1231464035809 } on : shard0001 Timestamp(1, 47) 
			{ "a" : 369.1231464035809 } -->> { "a" : 400.9077816735953 } on : shard0001 Timestamp(1, 48) 
			{ "a" : 400.9077816735953 } -->> { "a" : 428.33575583063066 } on : shard0001 Timestamp(1, 43) 
			{ "a" : 428.33575583063066 } -->> { "a" : 464.3598909024149 } on : shard0001 Timestamp(1, 44) 
			{ "a" : 464.3598909024149 } -->> { "a" : 485.9499386511743 } on : shard0001 Timestamp(1, 69) 
			{ "a" : 485.9499386511743 } -->> { "a" : 509.9674719385803 } on : shard0001 Timestamp(1, 70) 
			{ "a" : 509.9674719385803 } -->> { "a" : 533.4743619896472 } on : shard0001 Timestamp(1, 59) 
			{ "a" : 533.4743619896472 } -->> { "a" : 560.6943164020777 } on : shard0001 Timestamp(1, 60) 
			{ "a" : 560.6943164020777 } -->> { "a" : 585.3555612266064 } on : shard0001 Timestamp(1, 55) 
			{ "a" : 585.3555612266064 } -->> { "a" : 614.1663421876729 } on : shard0001 Timestamp(1, 56) 
			{ "a" : 614.1663421876729 } -->> { "a" : 639.1101200133562 } on : shard0001 Timestamp(1, 51) 
			{ "a" : 639.1101200133562 } -->> { "a" : 676.0435830801725 } on : shard0001 Timestamp(1, 52) 
			{ "a" : 676.0435830801725 } -->> { "a" : 705.7812218554318 } on : shard0001 Timestamp(1, 45) 
			{ "a" : 705.7812218554318 } -->> { "a" : 740.9445280209184 } on : shard0001 Timestamp(1, 46) 
			{ "a" : 740.9445280209184 } -->> { "a" : 776.5613903757185 } on : shard0001 Timestamp(1, 39) 
			{ "a" : 776.5613903757185 } -->> { "a" : 795.5897776409984 } on : shard0001 Timestamp(1, 73) 
			{ "a" : 795.5897776409984 } -->> { "a" : 814.5981046836823 } on : shard0001 Timestamp(1, 74) 
			{ "a" : 814.5981046836823 } -->> { "a" : 850.8794126100838 } on : shard0001 Timestamp(1, 37) 
			{ "a" : 850.8794126100838 } -->> { "a" : 872.0335685648024 } on : shard0001 Timestamp(1, 65) 
			{ "a" : 872.0335685648024 } -->> { "a" : 893.7056448776275 } on : shard0001 Timestamp(1, 66) 
			{ "a" : 893.7056448776275 } -->> { "a" : 913.5230726096779 } on : shard0001 Timestamp(1, 67) 
			{ "a" : 913.5230726096779 } -->> { "a" : 936.9124132208526 } on : shard0001 Timestamp(1, 68) 
			{ "a" : 936.9124132208526 } -->> { "a" : 965.9900616388768 } on : shard0001 Timestamp(1, 49) 
			{ "a" : 965.9900616388768 } -->> { "a" : 999.3081407155842 } on : shard0001 Timestamp(1, 50) 
			{ "a" : 999.3081407155842 } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 4) 
// ... snip ...
Count is 29999
assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:15)
    at doassert (src/mongo/shell/assert.js:6:5)
    at Function.assert.soon (src/mongo/shell/assert.js:110:60)
    at /mnt/slaves/Linux_32bit/mongo/jstests/sharding/mrShardedOutput.js:91:16
Wed Apr 24 17:06:14.085 JavaScript execution failed: assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds at src/mongo/shell/assert.js:L7
failed to load: /mnt/slaves/Linux_32bit/mongo/jstests/sharding/mrShardedOutput.js

Comment by Tad Marshall [ 24/Apr/13 ]

On Windows 64-bit:
http://buildbot.mongodb.org/builders/Windows%2064-bit/builds/5497/steps/test_11/logs/stdio
http://buildlogs.mongodb.org/Windows%2064-bit/builds/5497/test/sharding/mrShardedOutput.js

Setup FAILURE: testDB.foo.find().itcount() = 199999 after saving 200000 documents -- (will assert after diagnostics)
// ... snip ...
*** ====> mongos could not find document 153074 with shard key 326.5303687658161: not found on either shard
// ... snip ...
========> Finished checking 200000
 
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"version" : 3,
	"minCompatibleVersion" : 3,
	"currentVersion" : 4,
	"clusterId" : ObjectId("5177f177a007f69a8f8f3ca9")
}
  shards:
	{  "_id" : "shard0000",  "host" : "localhost:30000" }
	{  "_id" : "shard0001",  "host" : "localhost:30001" }
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard0001" }
		test.foo
			shard key: { "a" : 1 }
			chunks:
				shard0000	102
				shard0001	186
			{ "a" : { "$minKey" : 1 } } -->> { "a" : 0.46192528679966927 } on : shard0000 Timestamp(2, 0) 
			{ "a" : 0.46192528679966927 } -->> { "a" : 3.1429145019501448 } on : shard0000 Timestamp(46, 28) 
			{ "a" : 3.1429145019501448 } -->> { "a" : 5.901285680010915 } on : shard0000 Timestamp(46, 29) 
			{ "a" : 5.901285680010915 } -->> { "a" : 9.186218027025461 } on : shard0000 Timestamp(40, 54) 
			{ "a" : 9.186218027025461 } -->> { "a" : 13.362562516704202 } on : shard0000 Timestamp(40, 55) 
			{ "a" : 13.362562516704202 } -->> { "a" : 16.63308683782816 } on : shard0000 Timestamp(40, 58) 
			{ "a" : 16.63308683782816 } -->> { "a" : 21.377885015681386 } on : shard0000 Timestamp(40, 59) 
			{ "a" : 21.377885015681386 } -->> { "a" : 25.393693707883358 } on : shard0000 Timestamp(40, 32) 
			{ "a" : 25.393693707883358 } -->> { "a" : 29.724293388426304 } on : shard0000 Timestamp(40, 33) 
			{ "a" : 29.724293388426304 } -->> { "a" : 32.70293981768191 } on : shard0000 Timestamp(46, 10) 
			{ "a" : 32.70293981768191 } -->> { "a" : 37.59493841789663 } on : shard0000 Timestamp(46, 11) 
			{ "a" : 37.59493841789663 } -->> { "a" : 39.80945190414786 } on : shard0000 Timestamp(46, 124) 
			{ "a" : 39.80945190414786 } -->> { "a" : 42.178831063210964 } on : shard0000 Timestamp(46, 125) 
			{ "a" : 42.178831063210964 } -->> { "a" : 44.669229071587324 } on : shard0000 Timestamp(46, 118) 
			{ "a" : 44.669229071587324 } -->> { "a" : 47.2573796287179 } on : shard0000 Timestamp(46, 119) 
			{ "a" : 47.2573796287179 } -->> { "a" : 52.300314884632826 } on : shard0000 Timestamp(10, 8) 
			{ "a" : 52.300314884632826 } -->> { "a" : 57.000645669177175 } on : shard0000 Timestamp(10, 9) 
			{ "a" : 57.000645669177175 } -->> { "a" : 62.551094917580485 } on : shard0000 Timestamp(10, 4) 
			{ "a" : 62.551094917580485 } -->> { "a" : 65.4070780146867 } on : shard0000 Timestamp(43, 8) 
			{ "a" : 65.4070780146867 } -->> { "a" : 68.82614875212312 } on : shard0000 Timestamp(43, 9) 
			{ "a" : 68.82614875212312 } -->> { "a" : 73.24694562703371 } on : shard0000 Timestamp(38, 6) 
			{ "a" : 73.24694562703371 } -->> { "a" : 75.7027396466583 } on : shard0000 Timestamp(46, 76) 
			{ "a" : 75.7027396466583 } -->> { "a" : 78.87820340692997 } on : shard0000 Timestamp(46, 77) 
			{ "a" : 78.87820340692997 } -->> { "a" : 81.34629763662815 } on : shard0000 Timestamp(46, 56) 
			{ "a" : 81.34629763662815 } -->> { "a" : 84.91788990795612 } on : shard0000 Timestamp(46, 57) 
			{ "a" : 84.91788990795612 } -->> { "a" : 87.06079493276775 } on : shard0000 Timestamp(46, 106) 
			{ "a" : 87.06079493276775 } -->> { "a" : 91.11163020133972 } on : shard0000 Timestamp(46, 107) 
			{ "a" : 91.11163020133972 } -->> { "a" : 93.3444588445127 } on : shard0000 Timestamp(46, 96) 
			{ "a" : 93.3444588445127 } -->> { "a" : 96.77653852850199 } on : shard0000 Timestamp(46, 97) 
			{ "a" : 96.77653852850199 } -->> { "a" : 99.66547647491097 } on : shard0000 Timestamp(42, 8) 
			{ "a" : 99.66547647491097 } -->> { "a" : 103.01620722748339 } on : shard0000 Timestamp(42, 9) 
			{ "a" : 103.01620722748339 } -->> { "a" : 105.81168928183615 } on : shard0000 Timestamp(45, 2) 
			{ "a" : 105.81168928183615 } -->> { "a" : 108.83936425670981 } on : shard0000 Timestamp(45, 3) 
			{ "a" : 108.83936425670981 } -->> { "a" : 111.54887732118368 } on : shard0000 Timestamp(43, 2) 
			{ "a" : 111.54887732118368 } -->> { "a" : 114.96683536097407 } on : shard0000 Timestamp(43, 3) 
			{ "a" : 114.96683536097407 } -->> { "a" : 118.96160361357033 } on : shard0000 Timestamp(40, 34) 
			{ "a" : 118.96160361357033 } -->> { "a" : 121.75396899692714 } on : shard0000 Timestamp(45, 6) 
			{ "a" : 121.75396899692714 } -->> { "a" : 124.65395662002265 } on : shard0000 Timestamp(45, 7) 
			{ "a" : 124.65395662002265 } -->> { "a" : 127.3466448765248 } on : shard0000 Timestamp(43, 6) 
			{ "a" : 127.3466448765248 } -->> { "a" : 130.77009306289256 } on : shard0000 Timestamp(43, 7) 
			{ "a" : 130.77009306289256 } -->> { "a" : 133.35177348926663 } on : shard0000 Timestamp(46, 58) 
			{ "a" : 133.35177348926663 } -->> { "a" : 138.3365464862436 } on : shard0000 Timestamp(46, 59) 
			{ "a" : 138.3365464862436 } -->> { "a" : 141.43990818411112 } on : shard0000 Timestamp(43, 4) 
			{ "a" : 141.43990818411112 } -->> { "a" : 144.86041897907853 } on : shard0000 Timestamp(43, 5) 
			{ "a" : 144.86041897907853 } -->> { "a" : 148.69712269864976 } on : shard0000 Timestamp(40, 36) 
			{ "a" : 148.69712269864976 } -->> { "a" : 153.12424767762423 } on : shard0000 Timestamp(40, 37) 
			{ "a" : 153.12424767762423 } -->> { "a" : 156.2225252855569 } on : shard0000 Timestamp(42, 2) 
			{ "a" : 156.2225252855569 } -->> { "a" : 159.91029608994722 } on : shard0000 Timestamp(42, 3) 
			{ "a" : 159.91029608994722 } -->> { "a" : 162.8061686642468 } on : shard0000 Timestamp(41, 4) 
			{ "a" : 162.8061686642468 } -->> { "a" : 167.55035240203142 } on : shard0000 Timestamp(41, 5) 
			{ "a" : 167.55035240203142 } -->> { "a" : 171.56831524334848 } on : shard0000 Timestamp(40, 26) 
			{ "a" : 171.56831524334848 } -->> { "a" : 176.761029753834 } on : shard0000 Timestamp(40, 27) 
			{ "a" : 176.761029753834 } -->> { "a" : 181.59296573139727 } on : shard0000 Timestamp(21, 0) 
			{ "a" : 181.59296573139727 } -->> { "a" : 184.1482596937567 } on : shard0000 Timestamp(46, 82) 
			{ "a" : 184.1482596937567 } -->> { "a" : 187.6359994057566 } on : shard0000 Timestamp(46, 83) 
			{ "a" : 187.6359994057566 } -->> { "a" : 192.12980847805738 } on : shard0000 Timestamp(23, 0) 
			{ "a" : 192.12980847805738 } -->> { "a" : 194.56855789758265 } on : shard0000 Timestamp(46, 78) 
			{ "a" : 194.56855789758265 } -->> { "a" : 197.55421276204288 } on : shard0000 Timestamp(46, 79) 
			{ "a" : 197.55421276204288 } -->> { "a" : 200.1041262410581 } on : shard0000 Timestamp(46, 70) 
			{ "a" : 200.1041262410581 } -->> { "a" : 203.69230513460934 } on : shard0000 Timestamp(46, 71) 
			{ "a" : 203.69230513460934 } -->> { "a" : 206.42050611786544 } on : shard0000 Timestamp(46, 4) 
			{ "a" : 206.42050611786544 } -->> { "a" : 211.04152989573777 } on : shard0000 Timestamp(46, 5) 
			{ "a" : 211.04152989573777 } -->> { "a" : 213.64251896739006 } on : shard0000 Timestamp(46, 74) 
			{ "a" : 213.64251896739006 } -->> { "a" : 217.37686428241432 } on : shard0000 Timestamp(46, 75) 
			{ "a" : 217.37686428241432 } -->> { "a" : 220.84114304743707 } on : shard0000 Timestamp(40, 90) 
			{ "a" : 220.84114304743707 } -->> { "a" : 224.4546040892601 } on : shard0000 Timestamp(40, 91) 
			{ "a" : 224.4546040892601 } -->> { "a" : 228.0099391937256 } on : shard0000 Timestamp(40, 86) 
			{ "a" : 228.0099391937256 } -->> { "a" : 232.3436017613858 } on : shard0000 Timestamp(40, 87) 
			{ "a" : 232.3436017613858 } -->> { "a" : 236.69978766702116 } on : shard0000 Timestamp(40, 6) 
			{ "a" : 236.69978766702116 } -->> { "a" : 241.30295659415424 } on : shard0000 Timestamp(40, 7) 
			{ "a" : 241.30295659415424 } -->> { "a" : 244.9155484791845 } on : shard0000 Timestamp(41, 2) 
			{ "a" : 244.9155484791845 } -->> { "a" : 248.3714644331485 } on : shard0000 Timestamp(41, 3) 
			{ "a" : 248.3714644331485 } -->> { "a" : 250.68854843266308 } on : shard0000 Timestamp(46, 112) 
			{ "a" : 250.68854843266308 } -->> { "a" : 253.52103565819561 } on : shard0000 Timestamp(46, 113) 
			{ "a" : 253.52103565819561 } -->> { "a" : 255.75748458504677 } on : shard0000 Timestamp(46, 104) 
			{ "a" : 255.75748458504677 } -->> { "a" : 258.6720760446042 } on : shard0000 Timestamp(46, 105) 
			{ "a" : 258.6720760446042 } -->> { "a" : 262.1082600671798 } on : shard0000 Timestamp(40, 62) 
			{ "a" : 262.1082600671798 } -->> { "a" : 267.20536407083273 } on : shard0000 Timestamp(40, 63) 
			{ "a" : 267.20536407083273 } -->> { "a" : 271.69838547706604 } on : shard0000 Timestamp(40, 24) 
			{ "a" : 271.69838547706604 } -->> { "a" : 276.72786777839065 } on : shard0000 Timestamp(40, 25) 
			{ "a" : 276.72786777839065 } -->> { "a" : 280.04816197790205 } on : shard0000 Timestamp(40, 84) 
			{ "a" : 280.04816197790205 } -->> { "a" : 284.30585842579603 } on : shard0000 Timestamp(40, 85) 
			{ "a" : 284.30585842579603 } -->> { "a" : 286.6678377613425 } on : shard0000 Timestamp(46, 120) 
			{ "a" : 286.6678377613425 } -->> { "a" : 289.139935746789 } on : shard0000 Timestamp(46, 121) 
			{ "a" : 289.139935746789 } -->> { "a" : 291.7625275440514 } on : shard0000 Timestamp(46, 54) 
			{ "a" : 291.7625275440514 } -->> { "a" : 295.3668830450624 } on : shard0000 Timestamp(46, 55) 
			{ "a" : 295.3668830450624 } -->> { "a" : 297.67090757377446 } on : shard0000 Timestamp(46, 80) 
			{ "a" : 297.67090757377446 } -->> { "a" : 300.76661007478833 } on : shard0000 Timestamp(46, 81) 
			{ "a" : 300.76661007478833 } -->> { "a" : 304.8779696691781 } on : shard0000 Timestamp(40, 30) 
			{ "a" : 304.8779696691781 } -->> { "a" : 309.5727751497179 } on : shard0000 Timestamp(40, 31) 
			{ "a" : 309.5727751497179 } -->> { "a" : 312.17807973735034 } on : shard0000 Timestamp(46, 60) 
			{ "a" : 312.17807973735034 } -->> { "a" : 317.03071715310216 } on : shard0000 Timestamp(46, 61) 
			{ "a" : 317.03071715310216 } -->> { "a" : 319.2583955824375 } on : shard0000 Timestamp(46, 130) 
			{ "a" : 319.2583955824375 } -->> { "a" : 321.73936325125396 } on : shard0000 Timestamp(46, 131) 
			{ "a" : 321.73936325125396 } -->> { "a" : 324.75210377015173 } on : shard0000 Timestamp(44, 2) 
			{ "a" : 324.75210377015173 } -->> { "a" : 328.0431483872235 } on : shard0000 Timestamp(44, 3) 
			{ "a" : 328.0431483872235 } -->> { "a" : 331.4363674726337 } on : shard0000 Timestamp(44, 0) 
			{ "a" : 331.4363674726337 } -->> { "a" : 335.8906025532633 } on : shard0000 Timestamp(45, 0) 
			{ "a" : 335.8906025532633 } -->> { "a" : 340.3133791871369 } on : shard0000 Timestamp(46, 0) 
			{ "a" : 340.3133791871369 } -->> { "a" : 345.4259601421654 } on : shard0000 Timestamp(47, 0) 
			{ "a" : 345.4259601421654 } -->> { "a" : 348.6090279184282 } on : shard0000 Timestamp(48, 0) 
			{ "a" : 348.6090279184282 } -->> { "a" : 353.1235719565302 } on : shard0000 Timestamp(49, 0) 
			{ "a" : 353.1235719565302 } -->> { "a" : 357.6718515250832 } on : shard0001 Timestamp(49, 1) 
			{ "a" : 357.6718515250832 } -->> { "a" : 359.955252148211 } on : shard0001 Timestamp(46, 102) 
			{ "a" : 359.955252148211 } -->> { "a" : 362.8682184498757 } on : shard0001 Timestamp(46, 103) 
			{ "a" : 362.8682184498757 } -->> { "a" : 366.1106259096414 } on : shard0001 Timestamp(42, 6) 
			{ "a" : 366.1106259096414 } -->> { "a" : 370.0636674184352 } on : shard0001 Timestamp(42, 7) 
			{ "a" : 370.0636674184352 } -->> { "a" : 374.63444587774575 } on : shard0001 Timestamp(40, 2) 
			{ "a" : 374.63444587774575 } -->> { "a" : 379.0887638460845 } on : shard0001 Timestamp(40, 3) 
			{ "a" : 379.0887638460845 } -->> { "a" : 383.8032705243677 } on : shard0001 Timestamp(38, 20) 
			{ "a" : 383.8032705243677 } -->> { "a" : 386.1311811488122 } on : shard0001 Timestamp(46, 84) 
			{ "a" : 386.1311811488122 } -->> { "a" : 389.26496752537787 } on : shard0001 Timestamp(46, 85) 
			{ "a" : 389.26496752537787 } -->> { "a" : 393.852508161217 } on : shard0001 Timestamp(10, 32) 
			{ "a" : 393.852508161217 } -->> { "a" : 396.71097369864583 } on : shard0001 Timestamp(46, 42) 
			{ "a" : 396.71097369864583 } -->> { "a" : 400.18552215769887 } on : shard0001 Timestamp(46, 43) 
			{ "a" : 400.18552215769887 } -->> { "a" : 404.8889728728682 } on : shard0001 Timestamp(10, 42) 
			{ "a" : 404.8889728728682 } -->> { "a" : 407.57156000472605 } on : shard0001 Timestamp(46, 50) 
			{ "a" : 407.57156000472605 } -->> { "a" : 410.6903544161469 } on : shard0001 Timestamp(46, 51) 
			{ "a" : 410.6903544161469 } -->> { "a" : 415.1882566511631 } on : shard0001 Timestamp(38, 4) 
			{ "a" : 415.1882566511631 } -->> { "a" : 417.7805804647505 } on : shard0001 Timestamp(46, 86) 
			{ "a" : 417.7805804647505 } -->> { "a" : 420.6486090552062 } on : shard0001 Timestamp(46, 87) 
			{ "a" : 420.6486090552062 } -->> { "a" : 423.12275036238134 } on : shard0001 Timestamp(46, 46) 
			{ "a" : 423.12275036238134 } -->> { "a" : 425.97927595488727 } on : shard0001 Timestamp(46, 47) 
			{ "a" : 425.97927595488727 } -->> { "a" : 429.340178379789 } on : shard0001 Timestamp(40, 88) 
			{ "a" : 429.340178379789 } -->> { "a" : 432.7546732965857 } on : shard0001 Timestamp(40, 89) 
			{ "a" : 432.7546732965857 } -->> { "a" : 435.31475705094635 } on : shard0001 Timestamp(46, 66) 
			{ "a" : 435.31475705094635 } -->> { "a" : 438.4549835231155 } on : shard0001 Timestamp(46, 67) 
			{ "a" : 438.4549835231155 } -->> { "a" : 441.9638959225267 } on : shard0001 Timestamp(40, 60) 
			{ "a" : 441.9638959225267 } -->> { "a" : 445.81755599938333 } on : shard0001 Timestamp(40, 61) 
			{ "a" : 445.81755599938333 } -->> { "a" : 449.1187420208007 } on : shard0001 Timestamp(40, 92) 
			{ "a" : 449.1187420208007 } -->> { "a" : 452.9130384325981 } on : shard0001 Timestamp(40, 93) 
			{ "a" : 452.9130384325981 } -->> { "a" : 457.1138054598123 } on : shard0001 Timestamp(40, 22) 
			{ "a" : 457.1138054598123 } -->> { "a" : 461.41547174192965 } on : shard0001 Timestamp(40, 23) 
			{ "a" : 461.41547174192965 } -->> { "a" : 464.54810816794634 } on : shard0001 Timestamp(40, 78) 
			{ "a" : 464.54810816794634 } -->> { "a" : 468.8585875555873 } on : shard0001 Timestamp(40, 79) 
			{ "a" : 468.8585875555873 } -->> { "a" : 473.2449292205274 } on : shard0001 Timestamp(40, 4) 
			{ "a" : 473.2449292205274 } -->> { "a" : 478.2766276039183 } on : shard0001 Timestamp(40, 5) 
			{ "a" : 478.2766276039183 } -->> { "a" : 481.34841956198215 } on : shard0001 Timestamp(40, 94) 
			{ "a" : 481.34841956198215 } -->> { "a" : 485.4560608509928 } on : shard0001 Timestamp(40, 95) 
			{ "a" : 485.4560608509928 } -->> { "a" : 487.9418984055519 } on : shard0001 Timestamp(46, 122) 
			{ "a" : 487.9418984055519 } -->> { "a" : 490.3501982335001 } on : shard0001 Timestamp(46, 123) 
			{ "a" : 490.3501982335001 } -->> { "a" : 492.68759693950415 } on : shard0001 Timestamp(46, 94) 
			{ "a" : 492.68759693950415 } -->> { "a" : 495.79069181345403 } on : shard0001 Timestamp(46, 95) 
			{ "a" : 495.79069181345403 } -->> { "a" : 499.3532800581306 } on : shard0001 Timestamp(40, 68) 
			{ "a" : 499.3532800581306 } -->> { "a" : 503.81170283071697 } on : shard0001 Timestamp(40, 69) 
			{ "a" : 503.81170283071697 } -->> { "a" : 508.3339926786721 } on : shard0001 Timestamp(10, 20) 
			{ "a" : 508.3339926786721 } -->> { "a" : 511.20537146925926 } on : shard0001 Timestamp(46, 20) 
			{ "a" : 511.20537146925926 } -->> { "a" : 514.7314337082207 } on : shard0001 Timestamp(46, 21) 
			{ "a" : 514.7314337082207 } -->> { "a" : 517.2977906186134 } on : shard0001 Timestamp(46, 62) 
			{ "a" : 517.2977906186134 } -->> { "a" : 522.2491752356291 } on : shard0001 Timestamp(46, 63) 
			{ "a" : 522.2491752356291 } -->> { "a" : 525.9959644172341 } on : shard0001 Timestamp(40, 44) 
			{ "a" : 525.9959644172341 } -->> { "a" : 528.8722834084183 } on : shard0001 Timestamp(46, 6) 
			{ "a" : 528.8722834084183 } -->> { "a" : 532.0144887082279 } on : shard0001 Timestamp(46, 7) 
			{ "a" : 532.0144887082279 } -->> { "a" : 535.5991995893419 } on : shard0001 Timestamp(40, 40) 
			{ "a" : 535.5991995893419 } -->> { "a" : 540.2515125460923 } on : shard0001 Timestamp(40, 41) 
			{ "a" : 540.2515125460923 } -->> { "a" : 545.3798682428896 } on : shard0001 Timestamp(10, 28) 
			{ "a" : 545.3798682428896 } -->> { "a" : 548.1370431371033 } on : shard0001 Timestamp(46, 12) 
			{ "a" : 548.1370431371033 } -->> { "a" : 550.6972139701247 } on : shard0001 Timestamp(46, 13) 
			{ "a" : 550.6972139701247 } -->> { "a" : 555.0913014449179 } on : shard0001 Timestamp(10, 34) 
			{ "a" : 555.0913014449179 } -->> { "a" : 557.9701173119247 } on : shard0001 Timestamp(43, 10) 
			{ "a" : 557.9701173119247 } -->> { "a" : 561.1547334119678 } on : shard0001 Timestamp(43, 11) 
			{ "a" : 561.1547334119678 } -->> { "a" : 566.4027677848935 } on : shard0001 Timestamp(10, 24) 
			{ "a" : 566.4027677848935 } -->> { "a" : 569.314714288339 } on : shard0001 Timestamp(46, 26) 
			{ "a" : 569.314714288339 } -->> { "a" : 572.5770648568869 } on : shard0001 Timestamp(46, 27) 
			{ "a" : 572.5770648568869 } -->> { "a" : 577.3731837980449 } on : shard0001 Timestamp(10, 22) 
			{ "a" : 577.3731837980449 } -->> { "a" : 580.0773387309164 } on : shard0001 Timestamp(46, 14) 
			{ "a" : 580.0773387309164 } -->> { "a" : 583.5137066897005 } on : shard0001 Timestamp(46, 15) 
			{ "a" : 583.5137066897005 } -->> { "a" : 586.2421630881727 } on : shard0001 Timestamp(45, 4) 
			{ "a" : 586.2421630881727 } -->> { "a" : 589.8600229993463 } on : shard0001 Timestamp(45, 5) 
			{ "a" : 589.8600229993463 } -->> { "a" : 593.5925915837288 } on : shard0001 Timestamp(40, 38) 
			{ "a" : 593.5925915837288 } -->> { "a" : 598.257846198976 } on : shard0001 Timestamp(40, 39) 
			{ "a" : 598.257846198976 } -->> { "a" : 600.8722856640816 } on : shard0001 Timestamp(46, 52) 
			{ "a" : 600.8722856640816 } -->> { "a" : 604.3082079850137 } on : shard0001 Timestamp(46, 53) 
			{ "a" : 604.3082079850137 } -->> { "a" : 608.4098408464342 } on : shard0001 Timestamp(40, 28) 
			{ "a" : 608.4098408464342 } -->> { "a" : 613.2322361227125 } on : shard0001 Timestamp(40, 29) 
			{ "a" : 613.2322361227125 } -->> { "a" : 617.6065888721496 } on : shard0001 Timestamp(38, 8) 
			{ "a" : 617.6065888721496 } -->> { "a" : 619.9446378741413 } on : shard0001 Timestamp(46, 126) 
			{ "a" : 619.9446378741413 } -->> { "a" : 622.7943589910865 } on : shard0001 Timestamp(46, 127) 
			{ "a" : 622.7943589910865 } -->> { "a" : 625.5093335639685 } on : shard0001 Timestamp(46, 16) 
			{ "a" : 625.5093335639685 } -->> { "a" : 628.752019489184 } on : shard0001 Timestamp(46, 17) 
			{ "a" : 628.752019489184 } -->> { "a" : 632.201710017398 } on : shard0001 Timestamp(40, 74) 
			{ "a" : 632.201710017398 } -->> { "a" : 635.3940875269473 } on : shard0001 Timestamp(40, 75) 
			{ "a" : 635.3940875269473 } -->> { "a" : 639.7445208858699 } on : shard0001 Timestamp(39, 2) 
			{ "a" : 639.7445208858699 } -->> { "a" : 642.110814107582 } on : shard0001 Timestamp(46, 90) 
			{ "a" : 642.110814107582 } -->> { "a" : 645.4171373043209 } on : shard0001 Timestamp(46, 91) 
			{ "a" : 645.4171373043209 } -->> { "a" : 649.7657068539411 } on : shard0001 Timestamp(10, 16) 
			{ "a" : 649.7657068539411 } -->> { "a" : 652.7872886508703 } on : shard0001 Timestamp(42, 4) 
			{ "a" : 652.7872886508703 } -->> { "a" : 656.6488335374743 } on : shard0001 Timestamp(42, 5) 
			{ "a" : 656.6488335374743 } -->> { "a" : 659.3760012183338 } on : shard0001 Timestamp(46, 40) 
			{ "a" : 659.3760012183338 } -->> { "a" : 662.1743394061923 } on : shard0001 Timestamp(46, 41) 
			{ "a" : 662.1743394061923 } -->> { "a" : 665.6629198696464 } on : shard0001 Timestamp(40, 82) 
			{ "a" : 665.6629198696464 } -->> { "a" : 669.6974479127675 } on : shard0001 Timestamp(40, 83) 
			{ "a" : 669.6974479127675 } -->> { "a" : 672.6627890020609 } on : shard0001 Timestamp(45, 8) 
			{ "a" : 672.6627890020609 } -->> { "a" : 676.4855307992548 } on : shard0001 Timestamp(45, 9) 
			{ "a" : 676.4855307992548 } -->> { "a" : 681.318553397432 } on : shard0001 Timestamp(10, 6) 
			{ "a" : 681.318553397432 } -->> { "a" : 683.7324383668602 } on : shard0001 Timestamp(46, 132) 
			{ "a" : 683.7324383668602 } -->> { "a" : 686.6638395003974 } on : shard0001 Timestamp(46, 133) 
			{ "a" : 686.6638395003974 } -->> { "a" : 689.4850027747452 } on : shard0001 Timestamp(46, 44) 
			{ "a" : 689.4850027747452 } -->> { "a" : 692.6667848601937 } on : shard0001 Timestamp(46, 45) 
			{ "a" : 692.6667848601937 } -->> { "a" : 695.3040196094662 } on : shard0001 Timestamp(46, 8) 
			{ "a" : 695.3040196094662 } -->> { "a" : 699.0381635259837 } on : shard0001 Timestamp(46, 9) 
			{ "a" : 699.0381635259837 } -->> { "a" : 701.5754322055727 } on : shard0001 Timestamp(46, 72) 
			{ "a" : 701.5754322055727 } -->> { "a" : 704.8974549397826 } on : shard0001 Timestamp(46, 73) 
			{ "a" : 704.8974549397826 } -->> { "a" : 708.4674614015967 } on : shard0001 Timestamp(40, 46) 
			{ "a" : 708.4674614015967 } -->> { "a" : 712.5652395188808 } on : shard0001 Timestamp(40, 47) 
			{ "a" : 712.5652395188808 } -->> { "a" : 715.7613597810268 } on : shard0001 Timestamp(40, 70) 
			{ "a" : 715.7613597810268 } -->> { "a" : 720.3049035742879 } on : shard0001 Timestamp(40, 71) 
			{ "a" : 720.3049035742879 } -->> { "a" : 723.6232834402472 } on : shard0001 Timestamp(40, 56) 
			{ "a" : 723.6232834402472 } -->> { "a" : 728.4568957984447 } on : shard0001 Timestamp(40, 57) 
			{ "a" : 728.4568957984447 } -->> { "a" : 733.0544330179691 } on : shard0001 Timestamp(38, 14) 
			{ "a" : 733.0544330179691 } -->> { "a" : 735.5003815609962 } on : shard0001 Timestamp(46, 68) 
			{ "a" : 735.5003815609962 } -->> { "a" : 738.7608338613063 } on : shard0001 Timestamp(46, 69) 
			{ "a" : 738.7608338613063 } -->> { "a" : 743.3845750056207 } on : shard0001 Timestamp(10, 38) 
			{ "a" : 743.3845750056207 } -->> { "a" : 746.1124293040484 } on : shard0001 Timestamp(46, 38) 
			{ "a" : 746.1124293040484 } -->> { "a" : 749.5015666354448 } on : shard0001 Timestamp(46, 39) 
			{ "a" : 749.5015666354448 } -->> { "a" : 752.6978794485331 } on : shard0001 Timestamp(40, 76) 
			{ "a" : 752.6978794485331 } -->> { "a" : 756.6122433636338 } on : shard0001 Timestamp(40, 77) 
			{ "a" : 756.6122433636338 } -->> { "a" : 760.1049852091819 } on : shard0001 Timestamp(40, 64) 
			{ "a" : 760.1049852091819 } -->> { "a" : 764.8519398644567 } on : shard0001 Timestamp(40, 65) 
			{ "a" : 764.8519398644567 } -->> { "a" : 768.2935867924243 } on : shard0001 Timestamp(40, 42) 
			{ "a" : 768.2935867924243 } -->> { "a" : 771.1954885162413 } on : shard0001 Timestamp(45, 12) 
			{ "a" : 771.1954885162413 } -->> { "a" : 774.3603347335011 } on : shard0001 Timestamp(45, 13) 
			{ "a" : 774.3603347335011 } -->> { "a" : 777.0420722663403 } on : shard0001 Timestamp(46, 18) 
			{ "a" : 777.0420722663403 } -->> { "a" : 780.2039629314095 } on : shard0001 Timestamp(46, 19) 
			{ "a" : 780.2039629314095 } -->> { "a" : 782.800845336169 } on : shard0001 Timestamp(46, 30) 
			{ "a" : 782.800845336169 } -->> { "a" : 786.112894769758 } on : shard0001 Timestamp(46, 31) 
			{ "a" : 786.112894769758 } -->> { "a" : 790.4222474899143 } on : shard0001 Timestamp(40, 8) 
			{ "a" : 790.4222474899143 } -->> { "a" : 792.8061471320689 } on : shard0001 Timestamp(46, 110) 
			{ "a" : 792.8061471320689 } -->> { "a" : 795.8163765724748 } on : shard0001 Timestamp(46, 111) 
			{ "a" : 795.8163765724748 } -->> { "a" : 800.4844887182117 } on : shard0001 Timestamp(10, 26) 
			{ "a" : 800.4844887182117 } -->> { "a" : 803.2211179379374 } on : shard0001 Timestamp(46, 22) 
			{ "a" : 803.2211179379374 } -->> { "a" : 806.4851900562644 } on : shard0001 Timestamp(46, 23) 
			{ "a" : 806.4851900562644 } -->> { "a" : 811.2162575125694 } on : shard0001 Timestamp(38, 18) 
			{ "a" : 811.2162575125694 } -->> { "a" : 813.5626586154103 } on : shard0001 Timestamp(46, 100) 
			{ "a" : 813.5626586154103 } -->> { "a" : 816.5159979835153 } on : shard0001 Timestamp(46, 101) 
			{ "a" : 816.5159979835153 } -->> { "a" : 821.1802949663252 } on : shard0001 Timestamp(10, 12) 
			{ "a" : 821.1802949663252 } -->> { "a" : 823.6318798735738 } on : shard0001 Timestamp(46, 32) 
			{ "a" : 823.6318798735738 } -->> { "a" : 827.1924077998847 } on : shard0001 Timestamp(46, 33) 
			{ "a" : 827.1924077998847 } -->> { "a" : 829.7865479253232 } on : shard0001 Timestamp(46, 2) 
			{ "a" : 829.7865479253232 } -->> { "a" : 833.48797284998 } on : shard0001 Timestamp(46, 3) 
			{ "a" : 833.48797284998 } -->> { "a" : 835.8546982053667 } on : shard0001 Timestamp(46, 114) 
			{ "a" : 835.8546982053667 } -->> { "a" : 838.6866864748299 } on : shard0001 Timestamp(46, 115) 
			{ "a" : 838.6866864748299 } -->> { "a" : 841.0069236997515 } on : shard0001 Timestamp(46, 116) 
			{ "a" : 841.0069236997515 } -->> { "a" : 844.3400620017201 } on : shard0001 Timestamp(46, 117) 
			{ "a" : 844.3400620017201 } -->> { "a" : 848.2626746408641 } on : shard0001 Timestamp(40, 18) 
			{ "a" : 848.2626746408641 } -->> { "a" : 850.7346701808274 } on : shard0001 Timestamp(46, 92) 
			{ "a" : 850.7346701808274 } -->> { "a" : 854.1877407114953 } on : shard0001 Timestamp(46, 93) 
			{ "a" : 854.1877407114953 } -->> { "a" : 858.3319704048336 } on : shard0001 Timestamp(40, 10) 
			{ "a" : 858.3319704048336 } -->> { "a" : 860.7586806174368 } on : shard0001 Timestamp(46, 98) 
			{ "a" : 860.7586806174368 } -->> { "a" : 864.3624009564519 } on : shard0001 Timestamp(46, 99) 
			{ "a" : 864.3624009564519 } -->> { "a" : 868.5472207143903 } on : shard0001 Timestamp(40, 20) 
			{ "a" : 868.5472207143903 } -->> { "a" : 874.0357286296785 } on : shard0001 Timestamp(40, 21) 
			{ "a" : 874.0357286296785 } -->> { "a" : 878.5119398962706 } on : shard0001 Timestamp(40, 16) 
			{ "a" : 878.5119398962706 } -->> { "a" : 880.6736075785011 } on : shard0001 Timestamp(46, 128) 
			{ "a" : 880.6736075785011 } -->> { "a" : 883.7072718888521 } on : shard0001 Timestamp(46, 129) 
			{ "a" : 883.7072718888521 } -->> { "a" : 886.4571114536375 } on : shard0001 Timestamp(46, 36) 
			{ "a" : 886.4571114536375 } -->> { "a" : 889.8158872034401 } on : shard0001 Timestamp(46, 37) 
			{ "a" : 889.8158872034401 } -->> { "a" : 893.1232981849462 } on : shard0001 Timestamp(40, 66) 
			{ "a" : 893.1232981849462 } -->> { "a" : 896.5743980370462 } on : shard0001 Timestamp(40, 67) 
			{ "a" : 896.5743980370462 } -->> { "a" : 899.5074958074838 } on : shard0001 Timestamp(45, 10) 
			{ "a" : 899.5074958074838 } -->> { "a" : 903.0046591069549 } on : shard0001 Timestamp(45, 11) 
			{ "a" : 903.0046591069549 } -->> { "a" : 906.5234502777457 } on : shard0001 Timestamp(40, 48) 
			{ "a" : 906.5234502777457 } -->> { "a" : 910.5712396558374 } on : shard0001 Timestamp(40, 49) 
			{ "a" : 910.5712396558374 } -->> { "a" : 914.553085109219 } on : shard0001 Timestamp(40, 14) 
			{ "a" : 914.553085109219 } -->> { "a" : 919.6528848260641 } on : shard0001 Timestamp(40, 15) 
			{ "a" : 919.6528848260641 } -->> { "a" : 924.3079235311598 } on : shard0001 Timestamp(10, 10) 
			{ "a" : 924.3079235311598 } -->> { "a" : 926.6451392322779 } on : shard0001 Timestamp(46, 108) 
			{ "a" : 926.6451392322779 } -->> { "a" : 930.5251245386899 } on : shard0001 Timestamp(46, 109) 
			{ "a" : 930.5251245386899 } -->> { "a" : 935.4787308257073 } on : shard0001 Timestamp(10, 14) 
			{ "a" : 935.4787308257073 } -->> { "a" : 938.080815365538 } on : shard0001 Timestamp(46, 48) 
			{ "a" : 938.080815365538 } -->> { "a" : 941.1927396431565 } on : shard0001 Timestamp(46, 49) 
			{ "a" : 941.1927396431565 } -->> { "a" : 944.0306602045894 } on : shard0001 Timestamp(46, 34) 
			{ "a" : 944.0306602045894 } -->> { "a" : 947.5693495478481 } on : shard0001 Timestamp(46, 35) 
			{ "a" : 947.5693495478481 } -->> { "a" : 951.1411197017878 } on : shard0001 Timestamp(40, 50) 
			{ "a" : 951.1411197017878 } -->> { "a" : 954.6680490020663 } on : shard0001 Timestamp(40, 51) 
			{ "a" : 954.6680490020663 } -->> { "a" : 958.9578260201961 } on : shard0001 Timestamp(40, 12) 
			{ "a" : 958.9578260201961 } -->> { "a" : 964.3411072902381 } on : shard0001 Timestamp(40, 13) 
			{ "a" : 964.3411072902381 } -->> { "a" : 968.6145561281592 } on : shard0001 Timestamp(38, 16) 
			{ "a" : 968.6145561281592 } -->> { "a" : 970.9325809963048 } on : shard0001 Timestamp(46, 88) 
			{ "a" : 970.9325809963048 } -->> { "a" : 974.3660734966397 } on : shard0001 Timestamp(46, 89) 
			{ "a" : 974.3660734966397 } -->> { "a" : 979.1163245681673 } on : shard0001 Timestamp(10, 30) 
			{ "a" : 979.1163245681673 } -->> { "a" : 981.6370229236782 } on : shard0001 Timestamp(46, 24) 
			{ "a" : 981.6370229236782 } -->> { "a" : 985.0752423517406 } on : shard0001 Timestamp(46, 25) 
			{ "a" : 985.0752423517406 } -->> { "a" : 987.55321232602 } on : shard0001 Timestamp(46, 64) 
			{ "a" : 987.55321232602 } -->> { "a" : 991.0902213305235 } on : shard0001 Timestamp(46, 65) 
			{ "a" : 991.0902213305235 } -->> { "a" : 994.4024768192321 } on : shard0001 Timestamp(40, 80) 
			{ "a" : 994.4024768192321 } -->> { "a" : 998.1044437736273 } on : shard0001 Timestamp(40, 81) 
			{ "a" : 998.1044437736273 } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 4) 
// ... snip ...
Count is 199999
assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:15)
    at doassert (src/mongo/shell/assert.js:6:5)
    at Function.assert.soon (src/mongo/shell/assert.js:110:60)
    at D:\slave\Windows_64bit\mongo\jstests\sharding\mrShardedOutput.js:91:16
Wed Apr 24 10:57:24.133 JavaScript execution failed: assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds at src/mongo/shell/assert.js:L7
failed to load: D:\slave\Windows_64bit\mongo\jstests\sharding\mrShardedOutput.js

Comment by Tad Marshall [ 23/Apr/13 ]

On 2.4 Linux subscription RHEL build:
http://buildbot-special.10gen.com/builders/V2.4%20Linux%2064-bit%20Subscription%20RHEL%206.2/builds/49
http://buildbot-special.10gen.com/builders/V2.4%20Linux%2064-bit%20Subscription%20RHEL%206.2/builds/49/steps/shell_3/logs/stdio

========> Finished saving total of 200000 documents
// ... snip ...
----
No errors on insert batch.
----
// ... snip ...
----
Setup FAILURE: testDB.foo.find().itcount() = 199998 after saving 200000 documents -- (will assert after diagnostics)
----
// ... snip ...
*** ====> mongos could not find document 140756 with shard key 248.154042288661: not found on either shard
// ... snip ...
*** ====> mongos could not find document 140873 with shard key 247.74073623120785: not found on either shard
// ... snip ...
========> Finished checking 200000
 
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"version" : 3,
	"minCompatibleVersion" : 3,
	"currentVersion" : 4,
	"clusterId" : ObjectId("51767d682f11506ec2190015")
}
  shards:
	{  "_id" : "shard0000",  "host" : "localhost:30000" }
	{  "_id" : "shard0001",  "host" : "localhost:30001" }
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard0001" }
		test.foo
			shard key: { "a" : 1 }
			chunks:
				shard0000	79
				shard0001	201
			{ "a" : { "$minKey" : 1 } } -->> { "a" : 0.22516935132443905 } on : shard0000 { "t" : 2, "i" : 0 } 
			{ "a" : 0.22516935132443905 } -->> { "a" : 2.8838259167969227 } on : shard0000 { "t" : 37, "i" : 58 } 
			{ "a" : 2.8838259167969227 } -->> { "a" : 7.043199613690376 } on : shard0000 { "t" : 37, "i" : 59 } 
			{ "a" : 7.043199613690376 } -->> { "a" : 9.753296384587884 } on : shard0000 { "t" : 37, "i" : 50 } 
			{ "a" : 9.753296384587884 } -->> { "a" : 14.397597638890147 } on : shard0000 { "t" : 37, "i" : 51 } 
			{ "a" : 14.397597638890147 } -->> { "a" : 18.60163314267993 } on : shard0000 { "t" : 32, "i" : 4 } 
			{ "a" : 18.60163314267993 } -->> { "a" : 22.9238485917449 } on : shard0000 { "t" : 32, "i" : 5 } 
			{ "a" : 22.9238485917449 } -->> { "a" : 25.686740409582853 } on : shard0000 { "t" : 37, "i" : 48 } 
			{ "a" : 25.686740409582853 } -->> { "a" : 28.816535836085677 } on : shard0000 { "t" : 37, "i" : 49 } 
			{ "a" : 28.816535836085677 } -->> { "a" : 31.238643219694495 } on : shard0000 { "t" : 37, "i" : 188 } 
			{ "a" : 31.238643219694495 } -->> { "a" : 33.917819149792194 } on : shard0000 { "t" : 37, "i" : 189 } 
			{ "a" : 33.917819149792194 } -->> { "a" : 36.82484640739858 } on : shard0000 { "t" : 37, "i" : 2 } 
			{ "a" : 36.82484640739858 } -->> { "a" : 39.49638339690864 } on : shard0000 { "t" : 39, "i" : 2 } 
			{ "a" : 39.49638339690864 } -->> { "a" : 43.51714998483658 } on : shard0000 { "t" : 39, "i" : 3 } 
			{ "a" : 43.51714998483658 } -->> { "a" : 48.39965398423374 } on : shard0000 { "t" : 7, "i" : 76 } 
			{ "a" : 48.39965398423374 } -->> { "a" : 51.166669465601444 } on : shard0000 { "t" : 37, "i" : 78 } 
			{ "a" : 51.166669465601444 } -->> { "a" : 54.85819443129003 } on : shard0000 { "t" : 37, "i" : 79 } 
			{ "a" : 54.85819443129003 } -->> { "a" : 58.79963282495737 } on : shard0000 { "t" : 32, "i" : 12 } 
			{ "a" : 58.79963282495737 } -->> { "a" : 61.30597949959338 } on : shard0000 { "t" : 37, "i" : 108 } 
			{ "a" : 61.30597949959338 } -->> { "a" : 64.79135644622147 } on : shard0000 { "t" : 37, "i" : 109 } 
			{ "a" : 64.79135644622147 } -->> { "a" : 67.52411788329482 } on : shard0000 { "t" : 37, "i" : 160 } 
			{ "a" : 67.52411788329482 } -->> { "a" : 70.21470065228641 } on : shard0000 { "t" : 37, "i" : 161 } 
			{ "a" : 70.21470065228641 } -->> { "a" : 72.88195914588869 } on : shard0000 { "t" : 37, "i" : 126 } 
			{ "a" : 72.88195914588869 } -->> { "a" : 76.36059820652008 } on : shard0000 { "t" : 37, "i" : 127 } 
			{ "a" : 76.36059820652008 } -->> { "a" : 80.68597270175815 } on : shard0000 { "t" : 32, "i" : 8 } 
			{ "a" : 80.68597270175815 } -->> { "a" : 85.8563354704529 } on : shard0000 { "t" : 32, "i" : 9 } 
			{ "a" : 85.8563354704529 } -->> { "a" : 90.57442820630968 } on : shard0000 { "t" : 12, "i" : 0 } 
			{ "a" : 90.57442820630968 } -->> { "a" : 93.35964708589017 } on : shard0000 { "t" : 37, "i" : 94 } 
			{ "a" : 93.35964708589017 } -->> { "a" : 97.11910667829216 } on : shard0000 { "t" : 37, "i" : 95 } 
			{ "a" : 97.11910667829216 } -->> { "a" : 100.12807138264179 } on : shard0000 { "t" : 37, "i" : 146 } 
			{ "a" : 100.12807138264179 } -->> { "a" : 103.49807422608137 } on : shard0000 { "t" : 37, "i" : 147 } 
			{ "a" : 103.49807422608137 } -->> { "a" : 105.94551614485681 } on : shard0000 { "t" : 37, "i" : 110 } 
			{ "a" : 105.94551614485681 } -->> { "a" : 109.59871159866452 } on : shard0000 { "t" : 37, "i" : 111 } 
			{ "a" : 109.59871159866452 } -->> { "a" : 112.16164752840996 } on : shard0000 { "t" : 37, "i" : 128 } 
			{ "a" : 112.16164752840996 } -->> { "a" : 115.86475558578968 } on : shard0000 { "t" : 37, "i" : 129 } 
			{ "a" : 115.86475558578968 } -->> { "a" : 119.89255482330918 } on : shard0000 { "t" : 33, "i" : 4 } 
			{ "a" : 119.89255482330918 } -->> { "a" : 123.76261572353542 } on : shard0000 { "t" : 33, "i" : 5 } 
			{ "a" : 123.76261572353542 } -->> { "a" : 127.91789672337472 } on : shard0000 { "t" : 32, "i" : 18 } 
			{ "a" : 127.91789672337472 } -->> { "a" : 132.40574160590768 } on : shard0000 { "t" : 32, "i" : 19 } 
			{ "a" : 132.40574160590768 } -->> { "a" : 135.27398626320064 } on : shard0000 { "t" : 37, "i" : 164 } 
			{ "a" : 135.27398626320064 } -->> { "a" : 137.96697440557182 } on : shard0000 { "t" : 37, "i" : 165 } 
			{ "a" : 137.96697440557182 } -->> { "a" : 140.57452464476228 } on : shard0000 { "t" : 37, "i" : 64 } 
			{ "a" : 140.57452464476228 } -->> { "a" : 144.50334175489843 } on : shard0000 { "t" : 37, "i" : 65 } 
			{ "a" : 144.50334175489843 } -->> { "a" : 147.10238971747458 } on : shard0000 { "t" : 37, "i" : 162 } 
			{ "a" : 147.10238971747458 } -->> { "a" : 149.99034139327705 } on : shard0000 { "t" : 37, "i" : 163 } 
			{ "a" : 149.99034139327705 } -->> { "a" : 153.06038525886834 } on : shard0000 { "t" : 37, "i" : 36 } 
			{ "a" : 153.06038525886834 } -->> { "a" : 157.65069774352014 } on : shard0000 { "t" : 37, "i" : 37 } 
			{ "a" : 157.65069774352014 } -->> { "a" : 160.26546130888164 } on : shard0000 { "t" : 37, "i" : 52 } 
			{ "a" : 160.26546130888164 } -->> { "a" : 163.79321762360632 } on : shard0000 { "t" : 37, "i" : 53 } 
			{ "a" : 163.79321762360632 } -->> { "a" : 167.70128975622356 } on : shard0000 { "t" : 33, "i" : 2 } 
			{ "a" : 167.70128975622356 } -->> { "a" : 170.5257990397513 } on : shard0000 { "t" : 37, "i" : 184 } 
			{ "a" : 170.5257990397513 } -->> { "a" : 173.00544981844723 } on : shard0000 { "t" : 37, "i" : 185 } 
			{ "a" : 173.00544981844723 } -->> { "a" : 178.06240217760205 } on : shard0000 { "t" : 25, "i" : 0 } 
			{ "a" : 178.06240217760205 } -->> { "a" : 182.63330264016986 } on : shard0000 { "t" : 26, "i" : 0 } 
			{ "a" : 182.63330264016986 } -->> { "a" : 187.526993220672 } on : shard0000 { "t" : 27, "i" : 0 } 
			{ "a" : 187.526993220672 } -->> { "a" : 192.90926936082542 } on : shard0000 { "t" : 28, "i" : 0 } 
			{ "a" : 192.90926936082542 } -->> { "a" : 195.61646902002394 } on : shard0000 { "t" : 37, "i" : 74 } 
			{ "a" : 195.61646902002394 } -->> { "a" : 198.66833230480552 } on : shard0000 { "t" : 37, "i" : 75 } 
			{ "a" : 198.66833230480552 } -->> { "a" : 201.3340829871595 } on : shard0000 { "t" : 37, "i" : 172 } 
			{ "a" : 201.3340829871595 } -->> { "a" : 203.86892999522388 } on : shard0000 { "t" : 37, "i" : 173 } 
			{ "a" : 203.86892999522388 } -->> { "a" : 206.65034325793386 } on : shard0000 { "t" : 37, "i" : 158 } 
			{ "a" : 206.65034325793386 } -->> { "a" : 209.42416018806398 } on : shard0000 { "t" : 37, "i" : 159 } 
			{ "a" : 209.42416018806398 } -->> { "a" : 212.05672714859247 } on : shard0000 { "t" : 37, "i" : 120 } 
			{ "a" : 212.05672714859247 } -->> { "a" : 215.8747659996152 } on : shard0000 { "t" : 37, "i" : 121 } 
			{ "a" : 215.8747659996152 } -->> { "a" : 218.5705949086696 } on : shard0000 { "t" : 37, "i" : 168 } 
			{ "a" : 218.5705949086696 } -->> { "a" : 221.4181621093303 } on : shard0000 { "t" : 37, "i" : 169 } 
			{ "a" : 221.4181621093303 } -->> { "a" : 223.98345381952822 } on : shard0000 { "t" : 37, "i" : 174 } 
			{ "a" : 223.98345381952822 } -->> { "a" : 226.94076807238162 } on : shard0000 { "t" : 37, "i" : 175 } 
			{ "a" : 226.94076807238162 } -->> { "a" : 229.78684585541487 } on : shard0000 { "t" : 37, "i" : 30 } 
			{ "a" : 229.78684585541487 } -->> { "a" : 234.16170082055032 } on : shard0000 { "t" : 37, "i" : 31 } 
			{ "a" : 234.16170082055032 } -->> { "a" : 237.09707288071513 } on : shard0000 { "t" : 37, "i" : 22 } 
			{ "a" : 237.09707288071513 } -->> { "a" : 243.0915581062436 } on : shard0000 { "t" : 37, "i" : 23 } 
			{ "a" : 243.0915581062436 } -->> { "a" : 245.91211881488562 } on : shard0000 { "t" : 37, "i" : 82 } 
			{ "a" : 245.91211881488562 } -->> { "a" : 249.2967345751822 } on : shard0000 { "t" : 37, "i" : 83 } 
			{ "a" : 249.2967345751822 } -->> { "a" : 253.2233705278486 } on : shard0000 { "t" : 38, "i" : 0 } 
			{ "a" : 253.2233705278486 } -->> { "a" : 258.30924371257424 } on : shard0000 { "t" : 39, "i" : 0 } 
			{ "a" : 258.30924371257424 } -->> { "a" : 262.2299762442708 } on : shard0000 { "t" : 40, "i" : 0 } 
			{ "a" : 262.2299762442708 } -->> { "a" : 266.3961995858699 } on : shard0000 { "t" : 41, "i" : 0 } 
			{ "a" : 266.3961995858699 } -->> { "a" : 270.52238723263144 } on : shard0000 { "t" : 42, "i" : 0 } 
			{ "a" : 270.52238723263144 } -->> { "a" : 273.08457833714783 } on : shard0001 { "t" : 42, "i" : 1 } 
			{ "a" : 273.08457833714783 } -->> { "a" : 276.3588265515864 } on : shard0001 { "t" : 37, "i" : 155 } 
			{ "a" : 276.3588265515864 } -->> { "a" : 281.00779885426164 } on : shard0001 { "t" : 30, "i" : 6 } 
			{ "a" : 281.00779885426164 } -->> { "a" : 285.310146631673 } on : shard0001 { "t" : 30, "i" : 7 } 
			{ "a" : 285.310146631673 } -->> { "a" : 287.94682607986033 } on : shard0001 { "t" : 37, "i" : 86 } 
			{ "a" : 287.94682607986033 } -->> { "a" : 290.8635577186942 } on : shard0001 { "t" : 37, "i" : 87 } 
			{ "a" : 290.8635577186942 } -->> { "a" : 294.6965526789427 } on : shard0001 { "t" : 35, "i" : 6 } 
			{ "a" : 294.6965526789427 } -->> { "a" : 298.1934370473027 } on : shard0001 { "t" : 35, "i" : 7 } 
			{ "a" : 298.1934370473027 } -->> { "a" : 300.73085706681013 } on : shard0001 { "t" : 37, "i" : 116 } 
			{ "a" : 300.73085706681013 } -->> { "a" : 304.3574991170317 } on : shard0001 { "t" : 37, "i" : 117 } 
			{ "a" : 304.3574991170317 } -->> { "a" : 307.1810582187027 } on : shard0001 { "t" : 37, "i" : 26 } 
			{ "a" : 307.1810582187027 } -->> { "a" : 312.14772909879684 } on : shard0001 { "t" : 37, "i" : 27 } 
			{ "a" : 312.14772909879684 } -->> { "a" : 315.1473624166101 } on : shard0001 { "t" : 37, "i" : 46 } 
			{ "a" : 315.1473624166101 } -->> { "a" : 318.7175220809877 } on : shard0001 { "t" : 37, "i" : 47 } 
			{ "a" : 318.7175220809877 } -->> { "a" : 323.5389294568449 } on : shard0001 { "t" : 31, "i" : 6 } 
			{ "a" : 323.5389294568449 } -->> { "a" : 327.7912689372897 } on : shard0001 { "t" : 31, "i" : 7 } 
			{ "a" : 327.7912689372897 } -->> { "a" : 330.56483743712306 } on : shard0001 { "t" : 37, "i" : 122 } 
			{ "a" : 330.56483743712306 } -->> { "a" : 333.53888895362616 } on : shard0001 { "t" : 37, "i" : 123 } 
			{ "a" : 333.53888895362616 } -->> { "a" : 338.82038155570626 } on : shard0001 { "t" : 7, "i" : 49 } 
			{ "a" : 338.82038155570626 } -->> { "a" : 341.65356517769396 } on : shard0001 { "t" : 37, "i" : 28 } 
			{ "a" : 341.65356517769396 } -->> { "a" : 345.21293197758496 } on : shard0001 { "t" : 37, "i" : 29 } 
			{ "a" : 345.21293197758496 } -->> { "a" : 347.9823723901063 } on : shard0001 { "t" : 37, "i" : 38 } 
			{ "a" : 347.9823723901063 } -->> { "a" : 353.27181057073176 } on : shard0001 { "t" : 37, "i" : 39 } 
			{ "a" : 353.27181057073176 } -->> { "a" : 356.1711977235973 } on : shard0001 { "t" : 37, "i" : 60 } 
			{ "a" : 356.1711977235973 } -->> { "a" : 360.71756039746106 } on : shard0001 { "t" : 37, "i" : 61 } 
			{ "a" : 360.71756039746106 } -->> { "a" : 363.5033746249974 } on : shard0001 { "t" : 37, "i" : 20 } 
			{ "a" : 363.5033746249974 } -->> { "a" : 368.1959693785757 } on : shard0001 { "t" : 37, "i" : 21 } 
			{ "a" : 368.1959693785757 } -->> { "a" : 371.1274531669915 } on : shard0001 { "t" : 37, "i" : 34 } 
			{ "a" : 371.1274531669915 } -->> { "a" : 375.60159573331475 } on : shard0001 { "t" : 37, "i" : 35 } 
			{ "a" : 375.60159573331475 } -->> { "a" : 378.55182588100433 } on : shard0001 { "t" : 37, "i" : 4 } 
			{ "a" : 378.55182588100433 } -->> { "a" : 380.8421022258699 } on : shard0001 { "t" : 40, "i" : 2 } 
			{ "a" : 380.8421022258699 } -->> { "a" : 383.9607588015497 } on : shard0001 { "t" : 40, "i" : 3 } 
			{ "a" : 383.9607588015497 } -->> { "a" : 386.6678208578378 } on : shard0001 { "t" : 37, "i" : 68 } 
			{ "a" : 386.6678208578378 } -->> { "a" : 390.7962394878268 } on : shard0001 { "t" : 37, "i" : 69 } 
			{ "a" : 390.7962394878268 } -->> { "a" : 393.72068480588496 } on : shard0001 { "t" : 37, "i" : 40 } 
			{ "a" : 393.72068480588496 } -->> { "a" : 398.7824141513556 } on : shard0001 { "t" : 37, "i" : 41 } 
			{ "a" : 398.7824141513556 } -->> { "a" : 402.9432078823447 } on : shard0001 { "t" : 32, "i" : 10 } 
			{ "a" : 402.9432078823447 } -->> { "a" : 407.4307878036052 } on : shard0001 { "t" : 32, "i" : 11 } 
			{ "a" : 407.4307878036052 } -->> { "a" : 412.44234913028777 } on : shard0001 { "t" : 7, "i" : 84 } 
			{ "a" : 412.44234913028777 } -->> { "a" : 414.95443996973336 } on : shard0001 { "t" : 37, "i" : 140 } 
			{ "a" : 414.95443996973336 } -->> { "a" : 417.88076679222286 } on : shard0001 { "t" : 37, "i" : 141 } 
			{ "a" : 417.88076679222286 } -->> { "a" : 420.5967364832759 } on : shard0001 { "t" : 37, "i" : 62 } 
			{ "a" : 420.5967364832759 } -->> { "a" : 424.71024324186146 } on : shard0001 { "t" : 37, "i" : 63 } 
			{ "a" : 424.71024324186146 } -->> { "a" : 427.6078902184963 } on : shard0001 { "t" : 37, "i" : 10 } 
			{ "a" : 427.6078902184963 } -->> { "a" : 434.2415083665401 } on : shard0001 { "t" : 37, "i" : 11 } 
			{ "a" : 434.2415083665401 } -->> { "a" : 436.9719398673624 } on : shard0001 { "t" : 37, "i" : 176 } 
			{ "a" : 436.9719398673624 } -->> { "a" : 439.48654644191265 } on : shard0001 { "t" : 37, "i" : 177 } 
			{ "a" : 439.48654644191265 } -->> { "a" : 441.9485975522548 } on : shard0001 { "t" : 37, "i" : 142 } 
			{ "a" : 441.9485975522548 } -->> { "a" : 445.14051312580705 } on : shard0001 { "t" : 37, "i" : 143 } 
			{ "a" : 445.14051312580705 } -->> { "a" : 447.83902703784406 } on : shard0001 { "t" : 37, "i" : 90 } 
			{ "a" : 447.83902703784406 } -->> { "a" : 451.31643139757216 } on : shard0001 { "t" : 37, "i" : 91 } 
			{ "a" : 451.31643139757216 } -->> { "a" : 454.71551083028316 } on : shard0001 { "t" : 35, "i" : 12 } 
			{ "a" : 454.71551083028316 } -->> { "a" : 459.55675072036684 } on : shard0001 { "t" : 35, "i" : 13 } 
			{ "a" : 459.55675072036684 } -->> { "a" : 464.13167263381183 } on : shard0001 { "t" : 31, "i" : 12 } 
			{ "a" : 464.13167263381183 } -->> { "a" : 468.47290778532624 } on : shard0001 { "t" : 31, "i" : 13 } 
			{ "a" : 468.47290778532624 } -->> { "a" : 473.2024085242301 } on : shard0001 { "t" : 7, "i" : 70 } 
			{ "a" : 473.2024085242301 } -->> { "a" : 475.91438912786543 } on : shard0001 { "t" : 37, "i" : 100 } 
			{ "a" : 475.91438912786543 } -->> { "a" : 480.05273612216115 } on : shard0001 { "t" : 37, "i" : 101 } 
			{ "a" : 480.05273612216115 } -->> { "a" : 484.38143264502287 } on : shard0001 { "t" : 31, "i" : 10 } 
			{ "a" : 484.38143264502287 } -->> { "a" : 488.8758226297796 } on : shard0001 { "t" : 31, "i" : 11 } 
			{ "a" : 488.8758226297796 } -->> { "a" : 492.39113670773804 } on : shard0001 { "t" : 35, "i" : 10 } 
			{ "a" : 492.39113670773804 } -->> { "a" : 497.5240759085864 } on : shard0001 { "t" : 35, "i" : 11 } 
			{ "a" : 497.5240759085864 } -->> { "a" : 502.1973366383463 } on : shard0001 { "t" : 31, "i" : 8 } 
			{ "a" : 502.1973366383463 } -->> { "a" : 504.89066634327173 } on : shard0001 { "t" : 37, "i" : 150 } 
			{ "a" : 504.89066634327173 } -->> { "a" : 508.10328987427056 } on : shard0001 { "t" : 37, "i" : 151 } 
			{ "a" : 508.10328987427056 } -->> { "a" : 510.65561757422984 } on : shard0001 { "t" : 37, "i" : 92 } 
			{ "a" : 510.65561757422984 } -->> { "a" : 513.8545481022447 } on : shard0001 { "t" : 37, "i" : 93 } 
			{ "a" : 513.8545481022447 } -->> { "a" : 516.7734804563224 } on : shard0001 { "t" : 37, "i" : 118 } 
			{ "a" : 516.7734804563224 } -->> { "a" : 519.4515381008387 } on : shard0001 { "t" : 37, "i" : 119 } 
			{ "a" : 519.4515381008387 } -->> { "a" : 522.116901120171 } on : shard0001 { "t" : 37, "i" : 132 } 
			{ "a" : 522.116901120171 } -->> { "a" : 525.6040450185537 } on : shard0001 { "t" : 37, "i" : 133 } 
			{ "a" : 525.6040450185537 } -->> { "a" : 529.2296095285565 } on : shard0001 { "t" : 35, "i" : 4 } 
			{ "a" : 529.2296095285565 } -->> { "a" : 534.1079705394804 } on : shard0001 { "t" : 35, "i" : 5 } 
			{ "a" : 534.1079705394804 } -->> { "a" : 536.8853558320552 } on : shard0001 { "t" : 37, "i" : 136 } 
			{ "a" : 536.8853558320552 } -->> { "a" : 540.8191597089171 } on : shard0001 { "t" : 37, "i" : 137 } 
			{ "a" : 540.8191597089171 } -->> { "a" : 543.44735853374 } on : shard0001 { "t" : 37, "i" : 54 } 
			{ "a" : 543.44735853374 } -->> { "a" : 547.2589677665383 } on : shard0001 { "t" : 37, "i" : 55 } 
			{ "a" : 547.2589677665383 } -->> { "a" : 550.1953435596079 } on : shard0001 { "t" : 37, "i" : 16 } 
			{ "a" : 550.1953435596079 } -->> { "a" : 554.9031188711524 } on : shard0001 { "t" : 37, "i" : 17 } 
			{ "a" : 554.9031188711524 } -->> { "a" : 559.2433328274637 } on : shard0001 { "t" : 32, "i" : 2 } 
			{ "a" : 559.2433328274637 } -->> { "a" : 562.0115706697106 } on : shard0001 { "t" : 37, "i" : 106 } 
			{ "a" : 562.0115706697106 } -->> { "a" : 564.9546615313739 } on : shard0001 { "t" : 37, "i" : 107 } 
			{ "a" : 564.9546615313739 } -->> { "a" : 568.6946378555149 } on : shard0001 { "t" : 35, "i" : 8 } 
			{ "a" : 568.6946378555149 } -->> { "a" : 572.273540077731 } on : shard0001 { "t" : 35, "i" : 9 } 
			{ "a" : 572.273540077731 } -->> { "a" : 577.2969601675868 } on : shard0001 { "t" : 7, "i" : 60 } 
			{ "a" : 577.2969601675868 } -->> { "a" : 580.0367903430015 } on : shard0001 { "t" : 37, "i" : 104 } 
			{ "a" : 580.0367903430015 } -->> { "a" : 583.0571153201163 } on : shard0001 { "t" : 37, "i" : 105 } 
			{ "a" : 583.0571153201163 } -->> { "a" : 585.8498010784388 } on : shard0001 { "t" : 37, "i" : 112 } 
			{ "a" : 585.8498010784388 } -->> { "a" : 589.1754003241658 } on : shard0001 { "t" : 37, "i" : 113 } 
			{ "a" : 589.1754003241658 } -->> { "a" : 591.9568908866495 } on : shard0001 { "t" : 37, "i" : 180 } 
			{ "a" : 591.9568908866495 } -->> { "a" : 594.6516988333315 } on : shard0001 { "t" : 37, "i" : 181 } 
			{ "a" : 594.6516988333315 } -->> { "a" : 597.3347588442266 } on : shard0001 { "t" : 37, "i" : 84 } 
			{ "a" : 597.3347588442266 } -->> { "a" : 600.8114637807012 } on : shard0001 { "t" : 37, "i" : 85 } 
			{ "a" : 600.8114637807012 } -->> { "a" : 603.5308265127242 } on : shard0001 { "t" : 37, "i" : 44 } 
			{ "a" : 603.5308265127242 } -->> { "a" : 607.858722563833 } on : shard0001 { "t" : 37, "i" : 45 } 
			{ "a" : 607.858722563833 } -->> { "a" : 612.6964488066733 } on : shard0001 { "t" : 7, "i" : 82 } 
			{ "a" : 612.6964488066733 } -->> { "a" : 617.5161881837994 } on : shard0001 { "t" : 7, "i" : 83 } 
			{ "a" : 617.5161881837994 } -->> { "a" : 620.1698114164174 } on : shard0001 { "t" : 37, "i" : 170 } 
			{ "a" : 620.1698114164174 } -->> { "a" : 622.6216109935194 } on : shard0001 { "t" : 37, "i" : 171 } 
			{ "a" : 622.6216109935194 } -->> { "a" : 625.5523404106498 } on : shard0001 { "t" : 37, "i" : 42 } 
			{ "a" : 625.5523404106498 } -->> { "a" : 630.1174324471503 } on : shard0001 { "t" : 37, "i" : 43 } 
			{ "a" : 630.1174324471503 } -->> { "a" : 632.8102461993694 } on : shard0001 { "t" : 37, "i" : 70 } 
			{ "a" : 632.8102461993694 } -->> { "a" : 636.2445440609008 } on : shard0001 { "t" : 37, "i" : 71 } 
			{ "a" : 636.2445440609008 } -->> { "a" : 639.0345639083534 } on : shard0001 { "t" : 37, "i" : 98 } 
			{ "a" : 639.0345639083534 } -->> { "a" : 642.6708460785449 } on : shard0001 { "t" : 37, "i" : 99 } 
			{ "a" : 642.6708460785449 } -->> { "a" : 645.2340870164335 } on : shard0001 { "t" : 37, "i" : 80 } 
			{ "a" : 645.2340870164335 } -->> { "a" : 649.7372828889638 } on : shard0001 { "t" : 37, "i" : 81 } 
			{ "a" : 649.7372828889638 } -->> { "a" : 652.7336968574673 } on : shard0001 { "t" : 37, "i" : 8 } 
			{ "a" : 652.7336968574673 } -->> { "a" : 658.909359946847 } on : shard0001 { "t" : 37, "i" : 9 } 
			{ "a" : 658.909359946847 } -->> { "a" : 661.4200037438422 } on : shard0001 { "t" : 37, "i" : 88 } 
			{ "a" : 661.4200037438422 } -->> { "a" : 664.5921301096678 } on : shard0001 { "t" : 37, "i" : 89 } 
			{ "a" : 664.5921301096678 } -->> { "a" : 667.0197492931038 } on : shard0001 { "t" : 37, "i" : 148 } 
			{ "a" : 667.0197492931038 } -->> { "a" : 670.2951779589057 } on : shard0001 { "t" : 37, "i" : 149 } 
			{ "a" : 670.2951779589057 } -->> { "a" : 673.1579795014113 } on : shard0001 { "t" : 37, "i" : 18 } 
			{ "a" : 673.1579795014113 } -->> { "a" : 676.8580521456897 } on : shard0001 { "t" : 37, "i" : 19 } 
			{ "a" : 676.8580521456897 } -->> { "a" : 681.9491037167609 } on : shard0001 { "t" : 7, "i" : 90 } 
			{ "a" : 681.9491037167609 } -->> { "a" : 686.9077284354717 } on : shard0001 { "t" : 7, "i" : 91 } 
			{ "a" : 686.9077284354717 } -->> { "a" : 690.6754097435623 } on : shard0001 { "t" : 35, "i" : 2 } 
			{ "a" : 690.6754097435623 } -->> { "a" : 694.2423011641949 } on : shard0001 { "t" : 35, "i" : 3 } 
			{ "a" : 694.2423011641949 } -->> { "a" : 698.1855644844472 } on : shard0001 { "t" : 33, "i" : 8 } 
			{ "a" : 698.1855644844472 } -->> { "a" : 702.7725619263947 } on : shard0001 { "t" : 33, "i" : 9 } 
			{ "a" : 702.7725619263947 } -->> { "a" : 707.4526760261506 } on : shard0001 { "t" : 7, "i" : 92 } 
			{ "a" : 707.4526760261506 } -->> { "a" : 712.3278502840549 } on : shard0001 { "t" : 7, "i" : 93 } 
			{ "a" : 712.3278502840549 } -->> { "a" : 717.1299627516419 } on : shard0001 { "t" : 7, "i" : 94 } 
			{ "a" : 717.1299627516419 } -->> { "a" : 722.2932754084468 } on : shard0001 { "t" : 7, "i" : 95 } 
			{ "a" : 722.2932754084468 } -->> { "a" : 725.1065282616764 } on : shard0001 { "t" : 37, "i" : 124 } 
			{ "a" : 725.1065282616764 } -->> { "a" : 727.9736106283963 } on : shard0001 { "t" : 37, "i" : 125 } 
			{ "a" : 727.9736106283963 } -->> { "a" : 731.9460003636777 } on : shard0001 { "t" : 33, "i" : 6 } 
			{ "a" : 731.9460003636777 } -->> { "a" : 737.2183878906071 } on : shard0001 { "t" : 33, "i" : 7 } 
			{ "a" : 737.2183878906071 } -->> { "a" : 742.0161471236497 } on : shard0001 { "t" : 7, "i" : 88 } 
			{ "a" : 742.0161471236497 } -->> { "a" : 746.8414353206754 } on : shard0001 { "t" : 7, "i" : 89 } 
			{ "a" : 746.8414353206754 } -->> { "a" : 750.8740886114538 } on : shard0001 { "t" : 31, "i" : 4 } 
			{ "a" : 750.8740886114538 } -->> { "a" : 755.3809664677829 } on : shard0001 { "t" : 31, "i" : 5 } 
			{ "a" : 755.3809664677829 } -->> { "a" : 759.8396914545447 } on : shard0001 { "t" : 31, "i" : 2 } 
			{ "a" : 759.8396914545447 } -->> { "a" : 764.2941544763744 } on : shard0001 { "t" : 31, "i" : 3 } 
			{ "a" : 764.2941544763744 } -->> { "a" : 769.2047206219286 } on : shard0001 { "t" : 7, "i" : 72 } 
			{ "a" : 769.2047206219286 } -->> { "a" : 771.8802159652114 } on : shard0001 { "t" : 37, "i" : 182 } 
			{ "a" : 771.8802159652114 } -->> { "a" : 774.86622473225 } on : shard0001 { "t" : 37, "i" : 183 } 
			{ "a" : 774.86622473225 } -->> { "a" : 779.9748165998608 } on : shard0001 { "t" : 7, "i" : 64 } 
			{ "a" : 779.9748165998608 } -->> { "a" : 785.2809252217412 } on : shard0001 { "t" : 7, "i" : 65 } 
			{ "a" : 785.2809252217412 } -->> { "a" : 787.9073375370353 } on : shard0001 { "t" : 37, "i" : 190 } 
			{ "a" : 787.9073375370353 } -->> { "a" : 790.562342852354 } on : shard0001 { "t" : 37, "i" : 191 } 
			{ "a" : 790.562342852354 } -->> { "a" : 793.1017633527517 } on : shard0001 { "t" : 37, "i" : 186 } 
			{ "a" : 793.1017633527517 } -->> { "a" : 795.8248879294842 } on : shard0001 { "t" : 37, "i" : 187 } 
			{ "a" : 795.8248879294842 } -->> { "a" : 798.694217344746 } on : shard0001 { "t" : 37, "i" : 14 } 
			{ "a" : 798.694217344746 } -->> { "a" : 803.2205591443926 } on : shard0001 { "t" : 37, "i" : 15 } 
			{ "a" : 803.2205591443926 } -->> { "a" : 807.5505390297621 } on : shard0001 { "t" : 31, "i" : 14 } 
			{ "a" : 807.5505390297621 } -->> { "a" : 810.4109764099121 } on : shard0001 { "t" : 37, "i" : 152 } 
			{ "a" : 810.4109764099121 } -->> { "a" : 813.6199663858861 } on : shard0001 { "t" : 37, "i" : 153 } 
			{ "a" : 813.6199663858861 } -->> { "a" : 816.2423477042466 } on : shard0001 { "t" : 37, "i" : 156 } 
			{ "a" : 816.2423477042466 } -->> { "a" : 819.1175945103168 } on : shard0001 { "t" : 37, "i" : 157 } 
			{ "a" : 819.1175945103168 } -->> { "a" : 821.8070552684367 } on : shard0001 { "t" : 37, "i" : 130 } 
			{ "a" : 821.8070552684367 } -->> { "a" : 824.8239983804524 } on : shard0001 { "t" : 37, "i" : 131 } 
			{ "a" : 824.8239983804524 } -->> { "a" : 829.9805072601885 } on : shard0001 { "t" : 7, "i" : 66 } 
			{ "a" : 829.9805072601885 } -->> { "a" : 832.7306224964559 } on : shard0001 { "t" : 37, "i" : 138 } 
			{ "a" : 832.7306224964559 } -->> { "a" : 836.1700682435185 } on : shard0001 { "t" : 37, "i" : 139 } 
			{ "a" : 836.1700682435185 } -->> { "a" : 840.3765445109457 } on : shard0001 { "t" : 32, "i" : 6 } 
			{ "a" : 840.3765445109457 } -->> { "a" : 844.8165147565305 } on : shard0001 { "t" : 32, "i" : 7 } 
			{ "a" : 844.8165147565305 } -->> { "a" : 849.732220871374 } on : shard0001 { "t" : 7, "i" : 74 } 
			{ "a" : 849.732220871374 } -->> { "a" : 852.5153589434922 } on : shard0001 { "t" : 37, "i" : 102 } 
			{ "a" : 852.5153589434922 } -->> { "a" : 856.1296239495277 } on : shard0001 { "t" : 37, "i" : 103 } 
			{ "a" : 856.1296239495277 } -->> { "a" : 858.5924683138728 } on : shard0001 { "t" : 37, "i" : 166 } 
			{ "a" : 858.5924683138728 } -->> { "a" : 861.9551877491176 } on : shard0001 { "t" : 37, "i" : 167 } 
			{ "a" : 861.9551877491176 } -->> { "a" : 864.6672570612282 } on : shard0001 { "t" : 37, "i" : 144 } 
			{ "a" : 864.6672570612282 } -->> { "a" : 867.9490447975695 } on : shard0001 { "t" : 37, "i" : 145 } 
			{ "a" : 867.9490447975695 } -->> { "a" : 870.5415604636073 } on : shard0001 { "t" : 37, "i" : 56 } 
			{ "a" : 870.5415604636073 } -->> { "a" : 875.0237277708948 } on : shard0001 { "t" : 37, "i" : 57 } 
			{ "a" : 875.0237277708948 } -->> { "a" : 879.2277195025235 } on : shard0001 { "t" : 34, "i" : 2 } 
			{ "a" : 879.2277195025235 } -->> { "a" : 883.4486345294863 } on : shard0001 { "t" : 34, "i" : 3 } 
			{ "a" : 883.4486345294863 } -->> { "a" : 887.7619402483106 } on : shard0001 { "t" : 30, "i" : 2 } 
			{ "a" : 887.7619402483106 } -->> { "a" : 892.1398676466197 } on : shard0001 { "t" : 30, "i" : 3 } 
			{ "a" : 892.1398676466197 } -->> { "a" : 896.9798390753567 } on : shard0001 { "t" : 7, "i" : 68 } 
			{ "a" : 896.9798390753567 } -->> { "a" : 899.8038931749761 } on : shard0001 { "t" : 37, "i" : 96 } 
			{ "a" : 899.8038931749761 } -->> { "a" : 904.082037974149 } on : shard0001 { "t" : 37, "i" : 97 } 
			{ "a" : 904.082037974149 } -->> { "a" : 906.6166011616588 } on : shard0001 { "t" : 37, "i" : 178 } 
			{ "a" : 906.6166011616588 } -->> { "a" : 909.7943142987788 } on : shard0001 { "t" : 37, "i" : 179 } 
			{ "a" : 909.7943142987788 } -->> { "a" : 912.4797524418682 } on : shard0001 { "t" : 37, "i" : 76 } 
			{ "a" : 912.4797524418682 } -->> { "a" : 916.6328683495522 } on : shard0001 { "t" : 37, "i" : 77 } 
			{ "a" : 916.6328683495522 } -->> { "a" : 919.398003956303 } on : shard0001 { "t" : 37, "i" : 114 } 
			{ "a" : 919.398003956303 } -->> { "a" : 922.900851815939 } on : shard0001 { "t" : 37, "i" : 115 } 
			{ "a" : 922.900851815939 } -->> { "a" : 926.6527476720512 } on : shard0001 { "t" : 32, "i" : 16 } 
			{ "a" : 926.6527476720512 } -->> { "a" : 931.568760657683 } on : shard0001 { "t" : 32, "i" : 17 } 
			{ "a" : 931.568760657683 } -->> { "a" : 934.3840139918029 } on : shard0001 { "t" : 37, "i" : 134 } 
			{ "a" : 934.3840139918029 } -->> { "a" : 937.6548696309328 } on : shard0001 { "t" : 37, "i" : 135 } 
			{ "a" : 937.6548696309328 } -->> { "a" : 940.2594231069088 } on : shard0001 { "t" : 37, "i" : 32 } 
			{ "a" : 940.2594231069088 } -->> { "a" : 945.2006607316434 } on : shard0001 { "t" : 37, "i" : 33 } 
			{ "a" : 945.2006607316434 } -->> { "a" : 948.024621931836 } on : shard0001 { "t" : 37, "i" : 66 } 
			{ "a" : 948.024621931836 } -->> { "a" : 952.8721091337502 } on : shard0001 { "t" : 37, "i" : 67 } 
			{ "a" : 952.8721091337502 } -->> { "a" : 955.4359647445381 } on : shard0001 { "t" : 37, "i" : 6 } 
			{ "a" : 955.4359647445381 } -->> { "a" : 957.8535496257246 } on : shard0001 { "t" : 39, "i" : 4 } 
			{ "a" : 957.8535496257246 } -->> { "a" : 961.9131472427398 } on : shard0001 { "t" : 39, "i" : 5 } 
			{ "a" : 961.9131472427398 } -->> { "a" : 964.6876819897443 } on : shard0001 { "t" : 37, "i" : 24 } 
			{ "a" : 964.6876819897443 } -->> { "a" : 969.5200624410063 } on : shard0001 { "t" : 37, "i" : 25 } 
			{ "a" : 969.5200624410063 } -->> { "a" : 972.326309652999 } on : shard0001 { "t" : 37, "i" : 12 } 
			{ "a" : 972.326309652999 } -->> { "a" : 978.6397414281964 } on : shard0001 { "t" : 37, "i" : 13 } 
			{ "a" : 978.6397414281964 } -->> { "a" : 983.1796674989164 } on : shard0001 { "t" : 30, "i" : 4 } 
			{ "a" : 983.1796674989164 } -->> { "a" : 987.3056039214134 } on : shard0001 { "t" : 30, "i" : 5 } 
			{ "a" : 987.3056039214134 } -->> { "a" : 992.4698502290994 } on : shard0001 { "t" : 7, "i" : 54 } 
			{ "a" : 992.4698502290994 } -->> { "a" : 995.2424396760762 } on : shard0001 { "t" : 37, "i" : 72 } 
			{ "a" : 995.2424396760762 } -->> { "a" : 999.7594223823398 } on : shard0001 { "t" : 37, "i" : 73 } 
			{ "a" : 999.7594223823398 } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 { "t" : 1, "i" : 16 } 

Comment by Tad Marshall [ 20/Apr/13 ]

The Mac debug builder is the king at failing on this test:
http://buildbot.mongodb.org/builders/OS%20X%2010.5%2064-bit%20DEBUG/builds/1950/steps/test_11/logs/stdio
http://buildlogs.mongodb.org/OS%20X%2010.5%2064-bit%20DEBUG/builds/1950/test/sharding/mrShardedOutput.js

----
Setup FAILURE: testDB.foo.find().itcount() = 99986 after saving 100000 documents -- (will assert after diagnostics)
----
// ... snip ...
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9437 with shard key 110.14251760207117: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9444 with shard key 7.363179232925177: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9455 with shard key 111.66475154459476: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9461 with shard key 101.5766232740134: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9465 with shard key 51.892705261707306: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9469 with shard key 31.713532749563456: not found on either shard
 
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9470 with shard key 46.537336660549045: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9478 with shard key 103.13572105951607: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9485 with shard key 12.03162525780499: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9489 with shard key 35.933065228164196: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9493 with shard key 2.929686801508069: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9495 with shard key 34.606494940817356: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9501 with shard key 36.2546865362674: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 m30999| range.universal(): 1
 
*** ====> mongos could not find document 9519 with shard key 61.57400528900325: not found on either shard
 
 m30999| range.universal(): 1
 m30999| range.universal(): 1
// ... snip ...
 m30999| range.universal(): 1
 
========> Finished checking 100000
 
 m30000| Sat Apr 20 15:31:12.910 [conn6] info PageFaultRetryableSection will not yield, already locked upon reaching
 m30000| Sat Apr 20 15:31:13.179 [conn6] command config.$cmd command: { group: { cond: { ns: "test.foo" }, key: { shard: 1.0 }, initial: { nChunks: 0.0 }, ns: "chunks", $reduce: function ( doc , out ){ out.nChunks++; } } } ntoreturn:1 keyUpdates:0 locks(micros) r:706215 reslen:167 706ms
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"version" : 3,
	"minCompatibleVersion" : 3,
	"currentVersion" : 4,
	"clusterId" : ObjectId("5172ea113ea87f0c3fe90493")
}
  shards:
	{  "_id" : "shard0000",  "host" : "localhost:30000" }
	{  "_id" : "shard0001",  "host" : "localhost:30001" }
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard0001" }
		test.foo
			shard key: { "a" : 1 }
			chunks:
				shard0000	52
				shard0001	50
			{ "a" : { "$minKey" : 1 } } -->> { "a" : 0.13008015230298042 } on : shard0000 Timestamp(2, 0) 
			{ "a" : 0.13008015230298042 } -->> { "a" : 5.780557170510292 } on : shard0000 Timestamp(15, 8) 
			{ "a" : 5.780557170510292 } -->> { "a" : 19.22334684059024 } on : shard0000 Timestamp(15, 9) 
			{ "a" : 19.22334684059024 } -->> { "a" : 28.26341660693288 } on : shard0000 Timestamp(13, 42) 
			{ "a" : 28.26341660693288 } -->> { "a" : 45.75236374512315 } on : shard0000 Timestamp(13, 43) 
			{ "a" : 45.75236374512315 } -->> { "a" : 60.76132017187774 } on : shard0000 Timestamp(13, 4) 
			{ "a" : 60.76132017187774 } -->> { "a" : 65.52309263497591 } on : shard0000 Timestamp(15, 34) 
			{ "a" : 65.52309263497591 } -->> { "a" : 78.28037184663117 } on : shard0000 Timestamp(15, 35) 
			{ "a" : 78.28037184663117 } -->> { "a" : 94.52297585085034 } on : shard0000 Timestamp(13, 6) 
			{ "a" : 94.52297585085034 } -->> { "a" : 101.20240994729102 } on : shard0000 Timestamp(13, 70) 
			{ "a" : 101.20240994729102 } -->> { "a" : 117.38911387510598 } on : shard0000 Timestamp(13, 71) 
			{ "a" : 117.38911387510598 } -->> { "a" : 129.26566461101174 } on : shard0000 Timestamp(13, 24) 
			{ "a" : 129.26566461101174 } -->> { "a" : 144.2845412530005 } on : shard0000 Timestamp(13, 25) 
			{ "a" : 144.2845412530005 } -->> { "a" : 149.23656242899597 } on : shard0000 Timestamp(15, 22) 
			{ "a" : 149.23656242899597 } -->> { "a" : 163.557147141546 } on : shard0000 Timestamp(15, 23) 
			{ "a" : 163.557147141546 } -->> { "a" : 174.12347509525716 } on : shard0000 Timestamp(13, 34) 
			{ "a" : 174.12347509525716 } -->> { "a" : 179.61684963665903 } on : shard0000 Timestamp(15, 24) 
			{ "a" : 179.61684963665903 } -->> { "a" : 191.23613508418202 } on : shard0000 Timestamp(15, 25) 
			{ "a" : 191.23613508418202 } -->> { "a" : 206.93030720576644 } on : shard0000 Timestamp(13, 2) 
			{ "a" : 206.93030720576644 } -->> { "a" : 219.2211258225143 } on : shard0000 Timestamp(13, 18) 
			{ "a" : 219.2211258225143 } -->> { "a" : 224.21902045607567 } on : shard0000 Timestamp(15, 46) 
			{ "a" : 224.21902045607567 } -->> { "a" : 234.43695111200213 } on : shard0000 Timestamp(15, 47) 
			{ "a" : 234.43695111200213 } -->> { "a" : 246.93599622696638 } on : shard0000 Timestamp(13, 16) 
			{ "a" : 246.93599622696638 } -->> { "a" : 251.9578558858484 } on : shard0000 Timestamp(15, 26) 
			{ "a" : 251.9578558858484 } -->> { "a" : 260.218717623502 } on : shard0000 Timestamp(15, 27) 
			{ "a" : 260.218717623502 } -->> { "a" : 270.24244307540357 } on : shard0000 Timestamp(13, 36) 
			{ "a" : 270.24244307540357 } -->> { "a" : 286.1080018337816 } on : shard0000 Timestamp(13, 37) 
			{ "a" : 286.1080018337816 } -->> { "a" : 293.49741246551275 } on : shard0000 Timestamp(13, 62) 
			{ "a" : 293.49741246551275 } -->> { "a" : 305.5191889870912 } on : shard0000 Timestamp(13, 63) 
			{ "a" : 305.5191889870912 } -->> { "a" : 316.39757682569325 } on : shard0000 Timestamp(13, 26) 
			{ "a" : 316.39757682569325 } -->> { "a" : 328.6422551609576 } on : shard0000 Timestamp(13, 27) 
			{ "a" : 328.6422551609576 } -->> { "a" : 333.65489658899605 } on : shard0000 Timestamp(15, 16) 
			{ "a" : 333.65489658899605 } -->> { "a" : 346.2301343679428 } on : shard0000 Timestamp(15, 17) 
			{ "a" : 346.2301343679428 } -->> { "a" : 353.8862238638103 } on : shard0000 Timestamp(13, 60) 
			{ "a" : 353.8862238638103 } -->> { "a" : 366.2167901638895 } on : shard0000 Timestamp(13, 61) 
			{ "a" : 366.2167901638895 } -->> { "a" : 374.38753084279597 } on : shard0000 Timestamp(13, 46) 
			{ "a" : 374.38753084279597 } -->> { "a" : 379.61523444391787 } on : shard0000 Timestamp(15, 20) 
			{ "a" : 379.61523444391787 } -->> { "a" : 391.19666209444404 } on : shard0000 Timestamp(15, 21) 
			{ "a" : 391.19666209444404 } -->> { "a" : 401.26549056731164 } on : shard0000 Timestamp(13, 32) 
			{ "a" : 401.26549056731164 } -->> { "a" : 406.22597583569586 } on : shard0000 Timestamp(15, 36) 
			{ "a" : 406.22597583569586 } -->> { "a" : 416.3961026351899 } on : shard0000 Timestamp(15, 37) 
			{ "a" : 416.3961026351899 } -->> { "a" : 420.8605440799147 } on : shard0000 Timestamp(15, 44) 
			{ "a" : 420.8605440799147 } -->> { "a" : 432.3074482381344 } on : shard0000 Timestamp(15, 45) 
			{ "a" : 432.3074482381344 } -->> { "a" : 437.39596754312515 } on : shard0000 Timestamp(15, 28) 
			{ "a" : 437.39596754312515 } -->> { "a" : 448.5570606775582 } on : shard0000 Timestamp(15, 29) 
			{ "a" : 448.5570606775582 } -->> { "a" : 453.46290874294937 } on : shard0000 Timestamp(15, 32) 
			{ "a" : 453.46290874294937 } -->> { "a" : 465.82764480262995 } on : shard0000 Timestamp(15, 33) 
			{ "a" : 465.82764480262995 } -->> { "a" : 474.8192667029798 } on : shard0000 Timestamp(13, 44) 
			{ "a" : 474.8192667029798 } -->> { "a" : 479.7136893030256 } on : shard0000 Timestamp(15, 38) 
			{ "a" : 479.7136893030256 } -->> { "a" : 490.7433611806482 } on : shard0000 Timestamp(15, 39) 
			{ "a" : 490.7433611806482 } -->> { "a" : 499.344868818298 } on : shard0000 Timestamp(14, 0) 
			{ "a" : 499.344868818298 } -->> { "a" : 508.5908018518239 } on : shard0000 Timestamp(15, 0) 
			{ "a" : 508.5908018518239 } -->> { "a" : 516.9903580099344 } on : shard0001 Timestamp(15, 1) 
			{ "a" : 516.9903580099344 } -->> { "a" : 531.7791602574289 } on : shard0001 Timestamp(13, 51) 
			{ "a" : 531.7791602574289 } -->> { "a" : 544.0656463615596 } on : shard0001 Timestamp(13, 20) 
			{ "a" : 544.0656463615596 } -->> { "a" : 556.7231473978609 } on : shard0001 Timestamp(13, 21) 
			{ "a" : 556.7231473978609 } -->> { "a" : 565.7270224764943 } on : shard0001 Timestamp(13, 40) 
			{ "a" : 565.7270224764943 } -->> { "a" : 571.146473987028 } on : shard0001 Timestamp(15, 6) 
			{ "a" : 571.146473987028 } -->> { "a" : 582.7403762377799 } on : shard0001 Timestamp(15, 7) 
			{ "a" : 582.7403762377799 } -->> { "a" : 587.7366913482547 } on : shard0001 Timestamp(15, 40) 
			{ "a" : 587.7366913482547 } -->> { "a" : 598.3083816245198 } on : shard0001 Timestamp(15, 41) 
			{ "a" : 598.3083816245198 } -->> { "a" : 603.010349906981 } on : shard0001 Timestamp(15, 48) 
			{ "a" : 603.010349906981 } -->> { "a" : 615.1928449980915 } on : shard0001 Timestamp(15, 49) 
			{ "a" : 615.1928449980915 } -->> { "a" : 621.8069263268262 } on : shard0001 Timestamp(13, 64) 
			{ "a" : 621.8069263268262 } -->> { "a" : 633.3835741970688 } on : shard0001 Timestamp(13, 65) 
			{ "a" : 633.3835741970688 } -->> { "a" : 645.4779852647334 } on : shard0001 Timestamp(13, 22) 
			{ "a" : 645.4779852647334 } -->> { "a" : 657.995801884681 } on : shard0001 Timestamp(13, 23) 
			{ "a" : 657.995801884681 } -->> { "a" : 666.7160091456026 } on : shard0001 Timestamp(13, 56) 
			{ "a" : 666.7160091456026 } -->> { "a" : 677.2788988891989 } on : shard0001 Timestamp(13, 57) 
			{ "a" : 677.2788988891989 } -->> { "a" : 682.6010781805962 } on : shard0001 Timestamp(15, 4) 
			{ "a" : 682.6010781805962 } -->> { "a" : 698.7735612783581 } on : shard0001 Timestamp(15, 5) 
			{ "a" : 698.7735612783581 } -->> { "a" : 713.7446939013898 } on : shard0001 Timestamp(13, 8) 
			{ "a" : 713.7446939013898 } -->> { "a" : 719.2819176707417 } on : shard0001 Timestamp(15, 10) 
			{ "a" : 719.2819176707417 } -->> { "a" : 733.0715404823422 } on : shard0001 Timestamp(15, 11) 
			{ "a" : 733.0715404823422 } -->> { "a" : 738.0639561451972 } on : shard0001 Timestamp(15, 12) 
			{ "a" : 738.0639561451972 } -->> { "a" : 751.3990304432809 } on : shard0001 Timestamp(15, 13) 
			{ "a" : 751.3990304432809 } -->> { "a" : 766.4123128633946 } on : shard0001 Timestamp(13, 14) 
			{ "a" : 766.4123128633946 } -->> { "a" : 771.1178127210587 } on : shard0001 Timestamp(15, 42) 
			{ "a" : 771.1178127210587 } -->> { "a" : 781.6141115035862 } on : shard0001 Timestamp(15, 43) 
			{ "a" : 781.6141115035862 } -->> { "a" : 788.4157332591712 } on : shard0001 Timestamp(13, 66) 
			{ "a" : 788.4157332591712 } -->> { "a" : 801.2080329935998 } on : shard0001 Timestamp(13, 67) 
			{ "a" : 801.2080329935998 } -->> { "a" : 812.0440999045968 } on : shard0001 Timestamp(13, 30) 
			{ "a" : 812.0440999045968 } -->> { "a" : 826.9460622686893 } on : shard0001 Timestamp(13, 31) 
			{ "a" : 826.9460622686893 } -->> { "a" : 834.7824802622199 } on : shard0001 Timestamp(13, 58) 
			{ "a" : 834.7824802622199 } -->> { "a" : 846.6986720450222 } on : shard0001 Timestamp(13, 59) 
			{ "a" : 846.6986720450222 } -->> { "a" : 854.921210790053 } on : shard0001 Timestamp(13, 48) 
			{ "a" : 854.921210790053 } -->> { "a" : 868.4915967751294 } on : shard0001 Timestamp(13, 49) 
			{ "a" : 868.4915967751294 } -->> { "a" : 876.4984169974923 } on : shard0001 Timestamp(13, 54) 
			{ "a" : 876.4984169974923 } -->> { "a" : 887.3746360186487 } on : shard0001 Timestamp(13, 55) 
			{ "a" : 887.3746360186487 } -->> { "a" : 898.9696661010385 } on : shard0001 Timestamp(13, 28) 
			{ "a" : 898.9696661010385 } -->> { "a" : 903.8989185355604 } on : shard0001 Timestamp(15, 18) 
			{ "a" : 903.8989185355604 } -->> { "a" : 912.9002571571618 } on : shard0001 Timestamp(15, 19) 
			{ "a" : 912.9002571571618 } -->> { "a" : 921.545053133741 } on : shard0001 Timestamp(13, 52) 
			{ "a" : 921.545053133741 } -->> { "a" : 934.1793439816684 } on : shard0001 Timestamp(13, 53) 
			{ "a" : 934.1793439816684 } -->> { "a" : 939.6428954787552 } on : shard0001 Timestamp(15, 14) 
			{ "a" : 939.6428954787552 } -->> { "a" : 951.9889089278877 } on : shard0001 Timestamp(15, 15) 
			{ "a" : 951.9889089278877 } -->> { "a" : 957.4478284921497 } on : shard0001 Timestamp(15, 2) 
			{ "a" : 957.4478284921497 } -->> { "a" : 970.670185983181 } on : shard0001 Timestamp(15, 3) 
			{ "a" : 970.670185983181 } -->> { "a" : 979.6554988715798 } on : shard0001 Timestamp(13, 38) 
			{ "a" : 979.6554988715798 } -->> { "a" : 985.1429681293666 } on : shard0001 Timestamp(15, 30) 
			{ "a" : 985.1429681293666 } -->> { "a" : 997.98724334687 } on : shard0001 Timestamp(15, 31) 
			{ "a" : 997.98724334687 } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 4) 
 
 
 
----
Waiting for count to become correct
----
// ... snip ...
Count is 99986
assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:15)
    at doassert (src/mongo/shell/assert.js:6:5)
    at Function.assert.soon (src/mongo/shell/assert.js:110:60)
    at /Users/yellow/buildslaves/OS_X_105_64bit_DEBUG/mongo/jstests/sharding/mrShardedOutput.js:91:16
Sat Apr 20 15:31:46.340 JavaScript execution failed: assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds at src/mongo/shell/assert.js:L7
failed to load: /Users/yellow/buildslaves/OS_X_105_64bit_DEBUG/mongo/jstests/sharding/mrShardedOutput.js

Comment by Tad Marshall [ 05/Apr/13 ]

Still failing on master, and before it ran MapReduce:
http://buildbot-special.10gen.com/builders/Nightly%20Linux%2064-bit%20SSL%20RHEL%205.7/builds/301/steps/test_4/logs/stdio

----
Setup FAILURE: testDB.foo.find().itcount() = 99999 after saving 100000 documents -- (will assert after diagnostics)
----
// ... snip ...
========> Checked 93000
========> Checked 94000
========> Checked 95000
*** ====> mongos could not find document 95265 with shard key 29.897826490923762: not found on either shard
========> Checked 96000
========> Checked 97000
========> Checked 98000
========> Checked 99000
========> Finished checking 100000
// ... snip ...
  shards:
	{  "_id" : "shard0000",  "host" : "localhost:30000" }
	{  "_id" : "shard0001",  "host" : "localhost:30001" }
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard0001" }
		test.foo
			shard key: { "a" : 1 }
			chunks:
				shard0000	8
				shard0001	134
			{ "a" : { "$minKey" : 1 } } -->> { "a" : 0.11550635099411011 } on : shard0000 Timestamp(2, 0) 
			{ "a" : 0.11550635099411011 } -->> { "a" : 7.832858478650451 } on : shard0000 Timestamp(3, 0) 
			{ "a" : 7.832858478650451 } -->> { "a" : 17.27122371084988 } on : shard0000 Timestamp(4, 0) 
			{ "a" : 17.27122371084988 } -->> { "a" : 22.204731358215213 } on : shard0000 Timestamp(5, 0) 
			{ "a" : 22.204731358215213 } -->> { "a" : 28.665423160418868 } on : shard0000 Timestamp(6, 0) 
			{ "a" : 28.665423160418868 } -->> { "a" : 33.843352692201734 } on : shard0000 Timestamp(7, 0) 
			{ "a" : 33.843352692201734 } -->> { "a" : 40.68414308130741 } on : shard0000 Timestamp(8, 0) 
			{ "a" : 40.68414308130741 } -->> { "a" : 45.678226510062814 } on : shard0000 Timestamp(9, 0) 
			{ "a" : 45.678226510062814 } -->> { "a" : 50.96217757090926 } on : shard0001 Timestamp(9, 1) 
			{ "a" : 50.96217757090926 } -->> { "a" : 58.44867369160056 } on : shard0001 Timestamp(2, 60) 
			{ "a" : 58.44867369160056 } -->> { "a" : 66.53710105456412 } on : shard0001 Timestamp(2, 61) 
 

According to the map, this document should have been on shard0000.

Comment by Tad Marshall [ 08/Mar/13 ]

Ran test for 24 hours in a loop, completing 56 passes, on this machine without a failure. Restarted buildslave, and it failed again.
http://buildbot.mongodb.org/builders/OS%20X%2010.5%2064-bit%20DEBUG/builds/1864/steps/test_11/logs/stdio
http://buildlogs.mongodb.org/OS%20X%2010.5%2064-bit%20DEBUG/builds/1864/test/sharding/mrShardedOutput.js

*** ====> mongos could not find document 9945 with shard key 223.554011201486: not found on either shard

{ "a" : 210.50335722975433 } -->> { "a" : 223.81710237823427 } on : shard0000 { "t" : 7, "i" : 47 }

Count is 99999
assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:7)
    at doassert (src/mongo/shell/assert.js:6:1)
    at Function.assert.soon (src/mongo/shell/assert.js:110:48)
    at /Users/yellow/buildslaves/OS_X_105_64bit_DEBUG/mongo/jstests/sharding/mrShardedOutput.js:91:16
Fri Mar  8 01:38:44.699 JavaScript execution failed: assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds at src/mongo/shell/assert.js:L7
failed to load: /Users/yellow/buildslaves/OS_X_105_64bit_DEBUG/mongo/jstests/sharding/mrShardedOutput.js

Test hit "shard version not ok" around the time this document was received by shard0000:

 m30999| Fri Mar  8 01:22:20.947 [WriteBackListener-localhost:30001] writebacklisten result: { data: { writeBack: true, ns: "test.foo", connectionId: 3, instanceIdent: "bs-mm2.local:30001", version: Timestamp 4000|0, versionEpoch: ObjectId('5139835ecdff04ca91962aa8'), yourVersion: Timestamp 3000|1, yourVersionEpoch: ObjectId('5139835ecdff04ca91962aa8'), msg: BinData, id: ObjectId('5139839c00000000000001b8') }, ok: 1.0 }
 m30999| Fri Mar  8 01:22:20.947 [WriteBackListener-localhost:30001] connectionId: bs-mm2.local:30001:3 writebackId: 5139839c00000000000001b8 needVersion : 4|0||5139835ecdff04ca91962aa8 mine : 4|1||5139835ecdff04ca91962aa8
 m30999| Fri Mar  8 01:22:20.947 [WriteBackListener-localhost:30001] op: insert len: 1105 ns: test.foo{ _id: ObjectId('51398399964d03d5158f5c63'), a: 644.7675211820751, y: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...", i: 9942.0 }
 m30999| Fri Mar  8 01:22:20.947 [WriteBackListener-localhost:30001] wbl already reloaded config information for version 4|0||5139835ecdff04ca91962aa8, at version 4|1||5139835ecdff04ca91962aa8
 m30999| Fri Mar  8 01:22:20.955 [WriteBackListener-localhost:30001] writebacklisten result: { data: { writeBack: true, ns: "test.foo", connectionId: 3, instanceIdent: "bs-mm2.local:30001", version: Timestamp 4000|0, versionEpoch: ObjectId('5139835ecdff04ca91962aa8'), yourVersion: Timestamp 3000|1, yourVersionEpoch: ObjectId('5139835ecdff04ca91962aa8'), msg: BinData, id: ObjectId('5139839c00000000000001b9') }, ok: 1.0 }
 m30999| Fri Mar  8 01:22:20.955 [WriteBackListener-localhost:30001] connectionId: bs-mm2.local:30001:3 writebackId: 5139839c00000000000001b9 needVersion : 4|0||5139835ecdff04ca91962aa8 mine : 4|1||5139835ecdff04ca91962aa8
 m30999| Fri Mar  8 01:22:20.955 [WriteBackListener-localhost:30001] op: insert len: 1105 ns: test.foo{ _id: ObjectId('51398399964d03d5158f5c64'), a: 515.2052759658545, y: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...", i: 9943.0 }
 m30999| Fri Mar  8 01:22:20.955 [WriteBackListener-localhost:30001] wbl already reloaded config information for version 4|0||5139835ecdff04ca91962aa8, at version 4|1||5139835ecdff04ca91962aa8
 m30999| Fri Mar  8 01:22:20.956 [WriteBackListener-localhost:30001]     setShardVersion  shard0000 localhost:30000  test.foo  { setShardVersion: "test.foo", configdb: "localhost:30000", version: Timestamp 4000|0, versionEpoch: ObjectId('5139835ecdff04ca91962aa8'), serverID: ObjectId('5139835ccdff04ca91962aa6'), shard: "shard0000", shardHost: "localhost:30000" } 0x105b5afc0 20
 m30000| Fri Mar  8 01:22:21.019 [conn6] insert test.foo keyUpdates:0 exception: [test.foo] shard version not ok in Client::Context: version mismatch detected for test.foo, stored major version 4 does not match received 3 ( ns : test.foo, received : 3|0||5139835ecdff04ca91962aa8, wanted : 4|0||5139835ecdff04ca91962aa8, send ) code:13388 locks(micros) w:135 0ms
 m30999| Fri Mar  8 01:22:21.019 [WriteBackListener-localhost:30001]       setShardVersion success: { oldVersion: Timestamp 3000|0, oldVersionEpoch: ObjectId('5139835ecdff04ca91962aa8'), ok: 1.0 }
 m30000| Fri Mar  8 01:22:21.019 [conn4] command admin.$cmd command: { writebacklisten: ObjectId('5139835ccdff04ca91962aa6') } ntoreturn:1 keyUpdates:0  reslen:1362 62471ms
 m30999| Fri Mar  8 01:22:21.020 [WriteBackListener-localhost:30000] writebacklisten result: { data: { writeBack: true, ns: "test.foo", connectionId: 6, instanceIdent: "bs-mm2.local:30000", version: Timestamp 4000|0, versionEpoch: ObjectId('5139835ecdff04ca91962aa8'), yourVersion: Timestamp 3000|0, yourVersionEpoch: ObjectId('5139835ecdff04ca91962aa8'), msg: BinData, id: ObjectId('5139839d0000000000000000') }, ok: 1.0 }
 m30999| Fri Mar  8 01:22:21.020 [WriteBackListener-localhost:30000] connectionId: bs-mm2.local:30000:6 writebackId: 5139839d0000000000000000 needVersion : 4|0||5139835ecdff04ca91962aa8 mine : 4|1||5139835ecdff04ca91962aa8
 m30999| Fri Mar  8 01:22:21.020 [WriteBackListener-localhost:30000] op: insert len: 1105 ns: test.foo{ _id: ObjectId('51398399964d03d5158f5c69'), a: 348.4328999184072, y: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...", i: 9948.0 }
 m30999| Fri Mar  8 01:22:21.020 [WriteBackListener-localhost:30000] creating new connection to:localhost:30000
 m30999| Fri Mar  8 01:22:21.020 BackgroundJob starting: ConnectBG
 m30000| Fri Mar  8 01:22:21.021 [initandlisten] connection accepted from 127.0.0.1:62522 #18 (18 connections now open)
 m30999| Fri Mar  8 01:22:21.021 [WriteBackListener-localhost:30000] connected connection!

Comment by Tad Marshall [ 06/Mar/13 ]

Another failure:

http://buildbot.mongodb.org/builders/OS%20X%2010.5%2064-bit%20DEBUG/builds/1862/steps/test_11/logs/stdio
http://buildlogs.mongodb.org/OS%20X%2010.5%2064-bit%20DEBUG/builds/1862/test/sharding/mrShardedOutput.js

*** ====> mongos could not find document 2815 with shard key 45.83337320946157: not found on either shard

According to the chunk map, this document should have been on shard0001.

{ "a" : 43.28773496672511 } -->> { "a" : 60.713621554896235 } on : shard0001 { "t" : 7, "i" : 91 }

Count is 99999
assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:7)
    at doassert (src/mongo/shell/assert.js:6:1)
    at Function.assert.soon (src/mongo/shell/assert.js:110:48)
    at /Users/yellow/buildslaves/OS_X_105_64bit_DEBUG/mongo/jstests/sharding/mrShardedOutput.js:91:16
Wed Mar  6 00:32:10.194 JavaScript execution failed: assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds at src/mongo/shell/assert.js:L7
failed to load: /Users/yellow/buildslaves/OS_X_105_64bit_DEBUG/mongo/jstests/sharding/mrShardedOutput.js

Since it failed to find all inserted documents during the first pass, this run of this test never ran MapReduce.

Comment by auto [ 04/Mar/13 ]

Author:

{u'date': u'2013-03-04T20:28:42Z', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-8741 Add more diagnostics to test

Keep track of the shard key value (random number) for each saved
document and report it when a document can't be found so we can
see what shard it should be on and when it has been migrated.
When mongos can't find a document, look for it directly on both
shards and report where it was found or if it was not found.
Add header comments describing these features.
Branch: master
https://github.com/mongodb/mongo/commit/3f2de68c7ee94db972ee4a631208c3c8b0ee6f8f

Comment by Tad Marshall [ 02/Mar/13 ]

Happened on Mac OS/X 64-bit debug:
http://buildbot.mongodb.org/builders/OS%20X%2010.5%2064-bit%20DEBUG/builds/1857/steps/test_11/logs/stdio
http://buildlogs.mongodb.org/OS%20X%2010.5%2064-bit%20DEBUG/builds/1857/test/sharding/mrShardedOutput.js

...
----
Checking for missing documents
----
...
========> Could not find document 5941
...
----
Waiting for count to become correct
----
 
 
 m30999| Sat Mar  2 13:21:27.738 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Sat Mar  2 13:21:27.738 [Balancer] skipping balancing round because balancing is disabled
 m30999| Sat Mar  2 13:21:33.741 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Sat Mar  2 13:21:33.741 [Balancer] skipping balancing round because balancing is disabled
 m30999| Sat Mar  2 13:21:39.743 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Sat Mar  2 13:21:39.743 [Balancer] skipping balancing round because balancing is disabled
 m30999| Sat Mar  2 13:21:42.123 [LockPinger] cluster localhost:30000 pinged successfully at Sat Mar  2 13:21:42 2013 by distributed lock pinger 'localhost:30000/bs-mm2.local:30999:1362247480:16807', sleeping for 30000ms
 m30999| Sat Mar  2 13:21:45.745 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Sat Mar  2 13:21:45.745 [Balancer] skipping balancing round because balancing is disabled
Count is 99999
 m30999| Sat Mar  2 13:21:51.747 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Sat Mar  2 13:21:51.748 [Balancer] skipping balancing round because balancing is disabled
 m30999| Sat Mar  2 13:21:57.750 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Sat Mar  2 13:21:57.750 [Balancer] skipping balancing round because balancing is disabled
 m30999| Sat Mar  2 13:22:03.753 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Sat Mar  2 13:22:03.753 [Balancer] skipping balancing round because balancing is disabled
 m30999| Sat Mar  2 13:22:09.756 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Sat Mar  2 13:22:09.756 [Balancer] skipping balancing round because balancing is disabled
 m30999| Sat Mar  2 13:22:12.181 [LockPinger] cluster localhost:30000 pinged successfully at Sat Mar  2 13:22:12 2013 by distributed lock pinger 'localhost:30000/bs-mm2.local:30999:1362247480:16807', sleeping for 30000ms
 m30999| Sat Mar  2 13:22:15.758 [Balancer] Refreshing MaxChunkSize: 1
 m30999| Sat Mar  2 13:22:15.758 [Balancer] skipping balancing round because balancing is disabled
Count is 99999
assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds
Error: Printing Stack Trace
    at printStackTrace (src/mongo/shell/utils.js:37:7)
    at doassert (src/mongo/shell/assert.js:6:1)
    at Function.assert.soon (src/mongo/shell/assert.js:110:48)
    at /Users/yellow/buildslaves/OS_X_105_64bit_DEBUG/mongo/jstests/sharding/mrShardedOutput.js:76:16
Sat Mar  2 13:22:17.623 JavaScript execution failed: assert.soon failed: function () { var c = testDB.foo.find().itcount();
                                 print( "Count is " + c );
                                 return c == numDocs; }, msg:Setup FAILURE: Count did not become correct after 30 seconds at src/mongo/shell/assert.js:L7
failed to load: /Users/yellow/buildslaves/OS_X_105_64bit_DEBUG/mongo/jstests/sharding/mrShardedOutput.js

Comment by Tad Marshall [ 27/Feb/13 ]

One document really does seem to be missing. We're searching with the balancer stopped, so this should not be balancer-related.

 m30001| Tue Feb 26 08:48:52.506 [conn3] query test.foo query: { i: 112747.0 } ntoreturn:1 ntoskip:0 nscanned:149131 keyUpdates:0 locks(micros) r:111448 nreturned:0 reslen:20 111ms
 
========> Could not find document 112747
 
 m30001| Tue Feb 26 08:48:52.693 [conn3] query test.foo query: { i: 112748.0 } ntoreturn:1 ntoskip:0 nscanned:149131 keyUpdates:0 locks(micros) r:170704 nreturned:0 reslen:20 170ms

Comment by Daniel Pasette (Inactive) [ 27/Feb/13 ]

Tad, Can you put up a code review for the add index change you proposed in SERVER-8668?

Generated at Thu Feb 08 03:18:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.