[CSHARP-1510] GetMore throws an InvalidOperationException under certain conditions Created: 15/Dec/15  Updated: 02/Apr/16  Resolved: 07/Jan/16

Status: Closed
Project: C# Driver
Component/s: Operations
Affects Version/s: 2.2
Fix Version/s: 2.2.1

Type: Bug Priority: Major - P3
Reporter: Bret Ferrier Assignee: Robert Stam
Resolution: Done Votes: 0
Labels: regression
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows (Azure website and Win 10) .Net 4.6



 Description   

I have a query that is working against the 2.0 driver and have upgraded the driver to 2.2 and not updated the query and am getting the error below in some situations. I had to revert production to the 2.0 driver to get this to go away.

Below is the Error-

[InvalidOperationException: Nullable object must have a value.]
   System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource) +56
   System.Nullable`1.get_Value() +25
   MongoDB.Driver.Core.Operations.AsyncCursor`1.CalculateGetMoreProtocolNumberToReturn() +238
   MongoDB.Driver.Core.Operations.AsyncCursor`1.ExecuteGetMoreProtocolAsync(IChannelHandle channel, CancellationToken cancellationToken) +41
   MongoDB.Driver.Core.Operations.<GetNextBatchAsync>d__29.MoveNext() +1204
   System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13772500
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
   System.Runtime.CompilerServices.ConfiguredTaskAwaiter.GetResult() +35
   MongoDB.Driver.Core.Operations.<MoveNextAsync>d__32.MoveNext() +499
   System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13772500
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
   System.Runtime.CompilerServices.ConfiguredTaskAwaiter.GetResult() +31
   MongoDB.Driver.<ToListAsync>d__16`1.MoveNext() +660
   System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13772500
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
   System.Runtime.CompilerServices.ConfiguredTaskAwaiter.GetResult() +31
   MongoDB.Driver.<ToListAsync>d__16`1.MoveNext() +1006
   System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13772500
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
   System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() +31



 Comments   
Comment by Githook User [ 07/Jan/16 ]

Author:

{u'username': u'rstam', u'name': u'rstam', u'email': u'robert@robertstam.org'}

Message: CSHARP-1510: Fix GetMore throws InvalidOperationException under certain conditions.
Branch: v2.1.x
https://github.com/mongodb/mongo-csharp-driver/commit/ca8a8a1a957ca71ee5d32883ef3465207159bff9

Comment by Githook User [ 15/Dec/15 ]

Author:

{u'username': u'rstam', u'name': u'rstam', u'email': u'robert@robertstam.org'}

Message: CSHARP-1510: Fix GetMore throws InvalidOperationException under certain conditions.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/9a12e4a55adcd6c6a4f4f86140874d6963e74b43

Comment by Robert Stam [ 15/Dec/15 ]

Good point. We'll review the documentation.

BatchSize is in units of "documents". It is the maximum number of documents the server could return in a single batch.

Note: the server takes this as a hint and could return fewer documents.

Comment by Bret Ferrier [ 15/Dec/15 ]

So looking at the FindOptionsBase there is no documentation about what the units of the BatchSize are.

myIfindFluent.Options.BatchSize = 4; Is that 4KB or just 4 Bytes? I am assuming that setting it at 4MB is the best option till the fix is pushed out.

Also it seems that BatchSize is used in too many place as the documentation below uses it differently when talking about a Cursor and how many documents are returned and not size of the batch.
http://api.mongodb.org/csharp/current/html/P_MongoDB_Driver_MongoCursor_BatchSize.htm

Comment by Robert Stam [ 15/Dec/15 ]

The exact value for limit that will trigger this bug depends on the document sizes. The server returns batches of about 4MB in size, so whatever value for limit includes sufficient documents to exceed the 4MB batch size will trigger this bug.

You can work around this by setting the BatchSize to any non-null value. Avoid very small batch sizes as that would result in excessive round trips to the server.

Comment by Bret Ferrier [ 15/Dec/15 ]

Just did a test and found that I am unable to get a batch size of 600 but can get a batch size of 500 so the magic number is somewhere in between. Not sure if it is stagnant or dependent on the query.

Comment by Bret Ferrier [ 15/Dec/15 ]

Robert- What kind of Batch size is "too large" It is currently skipping 0 and setting a Batch size of 750 and is connecting to a 3.0 instance of Mongo.

Comment by Bret Ferrier [ 15/Dec/15 ]

If I change the code to use ToList as opposed to ToListAsync I get the following error

[InvalidOperationException: Nullable object must have a value.]
   System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource) +56
   System.Nullable`1.get_Value() +25
   MongoDB.Driver.Core.Operations.AsyncCursor`1.CalculateGetMoreProtocolNumberToReturn() +238
   MongoDB.Driver.Core.Operations.AsyncCursor`1.ExecuteGetMoreProtocol(IChannelHandle channel, CancellationToken cancellationToken) +48
   MongoDB.Driver.Core.Operations.AsyncCursor`1.GetNextBatch(CancellationToken cancellationToken) +182
   MongoDB.Driver.Core.Operations.AsyncCursor`1.MoveNext(CancellationToken cancellationToken) +90
   MongoDB.Driver.IAsyncCursorExtensions.ToList(IAsyncCursor`1 source, CancellationToken cancellationToken) +310
   MongoDB.Driver.IAsyncCursorSourceExtensions.ToList(IAsyncCursorSource`1 source, CancellationToken cancellationToken) +158

If I render out the Filter and the Sort Below is what I get
Filter

{
	"LongLat2" : {
		"$geoWithin" : {
			"$centerSphere" : [[-116.965858459473, 44.0256881713867], 0.012616093290458801]
		}
	},
	"When.DateStart" : {
		"$gte" : ISODate("2015-12-15T00:00:00Z"),
		"$lt" : ISODate("2016-12-14T00:00:00Z")
	},
	"ShareWith" : {
		"$in" : [2, 5267, 5312]
	},
	"BlockFrom" : {
		"$nin" : [2, 5267, 5312]
	}
}

Sort

		}
	},
	"When.DateStart" : {
		"$gte" : ISODate("2015-12-15T00:00:00Z"),
		"$lt" : ISODate("2016-12-14T00:00:00Z")
	},
	"ShareWith" : {
		"$in" : [2, 5267, 5312]
	},
	"BlockFrom" : {
		"$nin" : [2, 5267, 5312]
	}
}
 
Rendering out the Sort
{ "Popularity.8326" : -1, "PopularityIndex" : -1 }

Comment by Robert Stam [ 15/Dec/15 ]

Looks like the most likely way to encounter this is to use a very large limit with a default batch size of null.

The limit has to be large enough that the result set doesn't fit in the initial reply from the server.

Also, this should only occur on server versions < 3.2.

A workaround for now is to set a value for the batch size.

Comment by Robert Stam [ 15/Dec/15 ]

Thanks for reporting this. I will attempt to reproduce this.

If you can provide some code to reproduce this that would be very helpful.

Generated at Wed Feb 07 21:39:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.