InsertMany/BulkWrite/collection.Find fail with duplicated "incomplete read of message header: context deadline exceeded" error in both (v1 and v2)

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Works as Designed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • Go Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Detailed steps to reproduce the problem?

      1. Create a bath of records of 10kb and start insertMany in loop you will see this error in Go Lang

      Definition of done: what must be done to consider the task complete?

      The exact Go version used, with patch level:

      $ go version: 
      1.26.0-alpine3.23

      The exact version of the Go driver used:

      $ go list -m go.mongodb.org/mongo-driver
      go.mongodb.org/mongo-driver/v2 v2.8.0

      Describe how MongoDB is set up. Local vs Hosted, version, topology, load balanced, etc.

      The operating system and version (e.g. Windows 7, OSX 10.8, ...)

      Security Vulnerabilities

      NA with checkmarkx

       

       

      We are seeing repeated timeouts on InsertManyBulkWrite (upsert) and FindOne (findById-style) operations against a 3-node replica set with mTLS. The operation consistently fails
      at (or just past) the client-level operation timeout with a message that looks doubly wrapped, e.g. A: B: A: B:

      connection(host:27017[-2194]) incomplete read of message header:
      context deadline exceeded: client timed out waiting for server response:
      read tcp service_pod(kubernetes):52894->mongo(vm):27017: i/o timeout:
      connection(host:27017[-2194]) incomplete read of message header:
      context deadline exceeded: client timed out waiting for server response:
      read tcp service_pod(kubernetes)::52894->mongo(vm)::27017: i/o timeout

      We first observed this on the v1 driver, upgraded to v2 (go.mongodb.org/mongo-driver/v2 v2.8.0) hoping it was fixed, and see the same behaviour.

      RetryWrites and RetryReads are enabled, but the operation does not appear to be retried - it just returns after the operation timeout.

      Environment

      • Driver: go.mongodb.org/mongo-driver/v2 v2.8.0 (also reproduced on latest v1)
      • Go: 1.24.6
      • OS: Alpine Linux container on Kubernetes
      • Server: MongoDB replica set (3 nodes), mTLS enabled, client X.509 auth

      Client configuration (relevant options)

      clientOptions := options.Client().ApplyURI(uri) // mongodb://h1,h2,h3/?replicaSet=rs0&tls=trueclientOptions.SetBSONOptions(&options.BSONOptions

      {ObjectIDAsHexString: true}

      )clientOptions.SetTLSConfig(tlsCfg) // mTLS, session cache 128, RenegotiateNeverclientOptions.SetReplicaSet("rs0")

      clientOptions.SetMaxPoolSize(...) // e.g. 100clientOptions.SetMinPoolSize(...) // e.g. 10clientOptions.SetMaxConnIdleTime(...)
      clientOptions.SetConnectTimeout(...) // 10sclientOptions.SetTimeout(15 * time.Second) // client-level op timeoutclientOptions.SetServerSelectionTimeout(5 * time.Second)clientOptions.SetRetryWrites(true)clientOptions.SetRetryReads(true)clientOptions.SetMaxConnecting(5)clientOptions.SetServerMonitoringMode(options.ServerMonitoringModePoll)clientOptions.SetDialer(&net.Dialer

      {Timeout: 10*time.Second, KeepAlive: 15*time.Second}

      )clientOptions.SetPoolMonitor(...) // logs checkout-failed, pool-cleared, unexpected closeclientOptions.SetServerMonitor(...) // logs heartbeat-failed, description changesOps that fail (all use the standard Collection.* API):- Collection.InsertMany(ctx, docs)- Collection.BulkWrite(ctx, models) - upserts, ~200 items per batch- Collection.FindOne(ctx, bson.M

      {"_id": id}

      ).Decode(&out)Observed error (verbatim)

      [ERROR] [Kafka] Flush FAILED — partition=1, batchId=3c7f777f-906b-4510-84cc-303e65864ea6,elapsed=15.225997606s,err=bulkUpsertSending: connection(ie2p1xdlmd005.ie02.bpty:27017[-2194]) incomplete read of message header: context deadline exceeded: client timed out waiting for server response: read tcp 10.168.33.51:52894->10.168.83.194:27017: i/o timeout:connection(ie2p1xdlmd005.ie02.bpty:27017[-2194]) incomplete read of message header: context deadline exceeded: client timed out waiting for server response: read tcp service_pod(kubernetes):52894->mongo(vm):27017: i/o timeoutQuestions1. When readWireMessage returns incomplete read of message header: context deadline exceeded on a retryable op, is that classified as a retryable network error by the v2 driver? Our reading of the current code suggests it is treated as a context error and not retried - can you confirm?2. We are trying to insertMany with batch size of 10kb, 100kb, 400kb this is happening only when TLS is enabled but not happening with TLS disabled. Please check if this is something related to driver.

            Assignee:
            Preston Vasquez
            Reporter:
            Dikshit Thakral (EXT)
            None
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: