Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-14228

Setting batchSize and sort on a cursor in sharded collection causes fewer than all documents to be returned

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Duplicate
    • Affects Version/s: 2.4.8, 2.4.10
    • Fix Version/s: None
    • Component/s: Querying
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      This reproduces on a single instance:

      > db.foo.drop()
      		 
      > for (i=0; i<1000; i++) {db.foo.insert({x:i})}
      		 
      > db.foo.find().batchSize(100).itcount()
      		 
      1000
      		 
      > db.foo.find().sort({x:1}).batchSize(100).itcount()
      		 
      100

      Show
      This reproduces on a single instance: > db.foo.drop() > for (i=0; i<1000; i++) {db.foo.insert({x:i})} > db.foo.find().batchSize(100).itcount() 1000 > db.foo.find().sort({x:1}).batchSize(100).itcount() 100

      Description

      Here is the correct number of documents:

      mongos> db.ad_campaigns.find({fs:2}).count()
      		 
      4587

      If you specify a batchSize and a sort, you get fewer than all the documents:

      mongos> function cursorCount(query, batchSize) { var count = 0; var cursor = db.ad_campaigns.find(query).batchSize(batchSize); while(cursor.hasNext()) { cursor.next(); count++; }; return count }
      		 
      mongos> cursorCount({fs:2}, 100)
      		 
      3504

      The number of documents returned appears to be K + batchSize, for some K:

      mongos> cursorCount({fs:2}, 101)
      		 
      3505
      		 
      mongos> cursorCount({fs:2}, 102)
      		 
      3506

      K (3404 in this example) happens to match the number of documents matching the query on one shard (ads1):

      ads1> db.ad_campaigns.find({fs:2}).count()
      		 
      3404

      This relationship between K and the number of records on ads1 is borne out on other queries:

      mongos> db.ad_campaigns.find({fs:5}).count()
      		 
      9764
      		 
      mongos> cursorCount({fs:5}, 100)
      		 
      5249
      		 
      ads1> db.ad_campaigns.find({fs:5}).count()
      		 
      5149

      If the batch size is not specified or the cursor is not sorted, the problem goes away:

      mongos> db.ad_campaigns.find({fs:2}).count()
      		 
      4587
      		 
      mongos> cursorCount({fs:5}, 2000)
      		 
      4587
      		 
      mongos> var count = 0; var cursor = db.ad_campaigns.find({fs:2}).batchSize(100); while(cursor.hasNext()) { cursor.next(); count++; }; count
      		 
      4587

        Attachments

          Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. SERVER-12438 batch size with an unindexed sort in the new query system is inconsistent with the old behavior

          • Major - P3 - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-6015 using sort() with batchSize() never returns an active cursor, therefore incomplete resultsets can be returned

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SERVER-7267 Batch size + sort closes the cursor

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SERVER-14174 If ntoreturn is a limit (rather than batch size) extra data gets buffered during plan ranking

          • Major - P3 - Major loss of function.
          • Closed

          Question - SERVER-5374 batchSize is a hard limit for an in memory sort

          • Major - P3 - Major loss of function.
          • Closed

            Activity

              People

              Assignee:
              david.storch David Storch
              Reporter:
              jliszka Jason Liszka
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: