[CSHARP-4897] Index was outside the bounds of the array on changestream cursor Created: 02/Jan/24  Updated: 12/Jan/24  Resolved: 12/Jan/24

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ghazanfar Khan Assignee: Boris Dogadov
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2024-01-02-16-19-19-680.png    
Documentation Changes Summary:

1. What would you like to communicate to the user about this feature?
2. Would you like the user to see examples of the syntax and/or executable code and its output?
3. Which versions of the driver/connector does this apply to?


 Description   

Summary

We are using changestream using c# mongodb driver to do some realtime work. We received an error randomly after connection to the events.

$Index was outside the bounds of the array.

var options = new ChangeStreamOptions { ResumeAfter = resumeToken, BatchSize = 5 };

CancellationTokenSource cancelTokenSource = new CancellationTokenSource();
CancellationToken token = cancelTokenSource.Token;

try {
  using(var cursor = await mongoclient.WatchAsync(pipeline, options, cancellationToken: token)) {
    await cursor.ForEachAsync(async change =>

{            }

, cancellationToken: token);
  }
} catch (Exception ex) {

}

 

On this line 109 await cursor.ForEachAsync we receive this error. This happens after sometime like 8,9 hours every time.  Sharing the stacktrace

at Microsoft.Extensions.Logging.LogValuesFormatter.GetValue(Object[] values, Int32 index) at Microsoft.Extensions.Logging.FormattedLogValues.GetEnumerator()+MoveNext() at Serilog.Extensions.Logging.SerilogLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter) at Microsoft.Extensions.Logging.Logger`1.Microsoft.Extensions.Logging.ILogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter) at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, EventId eventId, Exception exception, String message, Object[] args) at SyncServer.Services.MultiDatabaseChangeStreamManager.<>cDisplayClass9_0.<<StartChangeStreams>b6>d.MoveNext() in /app/jenkins/workspace/flecx-prod-changestream/src/Host/SyncServer/Services/MultiDatabaseChangeStreamManager.cs:line 167 — End of stack trace from previous location — at MongoDB.Driver.IAsyncCursorExtensions.ForEachAsync[TDocument](IAsyncCursor`1 source, Func`3 processor, CancellationToken cancellationToken) at SyncServer.Services.MultiDatabaseChangeStreamManager.StartChangeStreams() in /app/jenkins/workspace/flecx-prod-changestream/src/Host/SyncServer/Services/MultiDatabaseChangeStreamManager.cs:line 109



 Comments   
Comment by Boris Dogadov [ 12/Jan/24 ]

Thank you for the update kghazanfar4@gmail.com 
I will close this ticket as resolved.

Comment by Ghazanfar Khan [ 12/Jan/24 ]

Hi, The issue have been resolved. Target replica set have members associated with domain names not with ips. We listed those ips in host files.

Comment by Boris Dogadov [ 08/Jan/24 ]

Hi kghazanfar4@gmail.com,

This exception points to a networking problem.
Driver can recover from transient networking issues, but if that causes any further issues I would suggest troubleshooting your network setup.

Thanks.

Comment by Ghazanfar Khan [ 07/Jan/24 ]

Hi thanks for reporting logs statement was throwing that exception. Now this is the exception that is raising

 

A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors = WritableServerSelector, LatencyLimitingServerSelector

{ AllowedLatencyRange = 00:00:00.0150000 }

, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : "2", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "

{ ClusterId : 2, EndPoint : "Unspecified/ibdxanv-cdb02.corp.ibexglobal.com:27017" }

", EndPoint: "Unspecified/ibdxanv-cdb02.corp.ibexglobal.com:27017", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 5.0.0, TopologyVersion: { "processId" : ObjectId("659524be65487d9bd258156d"), "counter" : NumberLong(1) }, Type: "ReplicaSetArbiter", WireVersionRange: "[0, 13]", LastHeartbeatTimestamp: "2024-01-06T20:58:29.1317025Z", LastUpdateTimestamp: "2024-01-06T20:58:29.1317027Z" }, { ServerId: "

{ ClusterId : 2, EndPoint : "Unspecified/ibdxanv-nosql02.corp.ibexglobal.com:27017" }

", EndPoint: "Unspecified/ibdxanv-nosql02.corp.ibexglobal.com:27017", ReasonChanged: "Heartbeat", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, ValueStopwatch stopwatch) at System.Net.Dns.GetHostAddresses(String hostNameOrAddress, AddressFamily family) at MongoDB.Driver.Core.Connections.TcpStreamFactory.ResolveEndPoints(EndPoint initial) at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.SslStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken) — End of inner exception stack trace — at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken cancellationToken) at MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnection(CancellationToken cancellationToken) at MongoDB.Driver.Core.Servers.ServerMonitor.Heartbeat(CancellationToken cancellationToken)", LastHeartbeatTimestamp: "2024-01-06T20:58:32.9818476Z", LastUpdateTimestamp: "2024-01-06T20:58:32.9818478Z" }, { ServerId: "

{ ClusterId : 2, EndPoint : "Unspecified/ibdxanv-nosql03.corp.ibexglobal.com:27017" }

", EndPoint: "Unspecified/ibdxanv-nosql03.corp.ibexglobal.com:27017", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 5.0.0, TopologyVersion: { "processId" : ObjectId("658d57cbcbad5cc5d47ebb13"), "counter" : NumberLong(3) }, Type: "ReplicaSetSecondary", WireVersionRange: "[0, 13]", LastHeartbeatTimestamp: "2024-01-06T20:58:31.0034211Z", LastUpdateTimestamp: "2024-01-06T20:58:31.0034216Z" }] }. 

Comment by Boris Dogadov [ 05/Jan/24 ]

Hi kghazanfar4@gmail.com 

It's still does not look like the failure is caused by the driver.
We would be able to investigate further if you could provide a stacktrace that points to driver code, and/or reproducible code sample.

Another thought, unrelated to C# driver, is that lines of code like:

 _logger.LogInformation($"ReplayChangeToTarget Exception :: {ex.Message}", ex.ToString());

Are not safe.  as if ex.Message contains quotes (for example ex.Message == "some text {id} more text"), they will be interpreted as a log parameter. And that would cause exactly the exception you are receiving.
Please also note that there is no placeholder for a parameter in the log template, yet a parameter is passed (ex.ToString()).

 

Comment by Ghazanfar Khan [ 05/Jan/24 ]

Hi,

Dotnet version = 6.0

Driver version = 2.20

OS = linux

 

I have added all the code in this post

 

https://stackoverflow.com/questions/77747804/mongodb-c-sharp-changestream-cursor-index-was-outside-the-bounds-of-the-array

 

We are receiving this error randomly.

Comment by Boris Dogadov [ 05/Jan/24 ]

Hi kghazanfar4@gmail.com 

I failed to find any evidence to an exception within the driver in the provided stacktrace 
it looks like the exception might be raised within the delegate passed to ForEachAsync.

To investigate further, could please share:
1. Driver, dotnet and OS versions being used.
2. A self contained repro.
3. Or a stacktrace pointing to a failure within the driver with a sample code.

Thanks.

Comment by Ghazanfar Khan [ 03/Jan/24 ]

dbeng-pm-bot  any update?

Comment by PM Bot [ 02/Jan/24 ]

Hi kghazanfar4@gmail.com, thank you for reporting this issue! The team will look into it and get back to you soon.

Generated at Wed Feb 07 21:49:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.