Uploaded image for project: 'Go Driver'
  1. Go Driver
  2. GODRIVER-3378

Partial write failure using InsertMany results in missing inserted ids

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Unknown Unknown
    • 2.0.1
    • Affects Version/s: None
    • Component/s: None
    • None
    • Go Drivers
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?

      Detailed steps to reproduce the problem?

      This occurs when the batch logic partitions data p1,p2,...,pn and fails when advancing pj where j > 1. AFAICT this only happens when one of the batches contains a document that is too large. That is, the error is not a WriteCommandError, checked here. You can use the following to reproduce this behavior:

      package main
      
      import (
      	"context"
      	"fmt"
      	"log"
      
      	"go.mongodb.org/mongo-driver/v2/bson"
      	"go.mongodb.org/mongo-driver/v2/mongo"
      )
      
      func main() {
      	client, err := mongo.Connect()
      	if err != nil {
      		panic(err)
      	}
      
      	defer func() { client.Disconnect(context.Background()) }()
      
      	coll := client.Database("db").Collection("coll")
      
      	docs := make([]any, 100)
      
      	for i := range docs {
      		docs[i] = bson.D{{"x", 1}}
      	}
      
      	docs = append(docs, bson.D{{"x", string(make([]byte, 16_777_150))}})
      	fmt.Println("len before adding breaker: ", len(docs))
      	docs = append(docs, bson.D{{"y", string(make([]byte, 17_000_000))}})
      
      	_, err = coll.InsertMany(context.Background(), docs)
      	if err != nil {
      		log.Println("err: ", err)
      	}
      
      	cur, err := coll.Find(context.Background(), bson.D{})
      	if err != nil {
      		log.Fatalf("failed to execute find command: %v", err)
      	}
      
      	var found []bson.D
      
      	if err := cur.All(context.Background(), &found); err != nil {
      		log.Fatalf("failed to decode data: %v", err)
      	}
      
      	fmt.Println("found: ", len(found))
      }
      

      Definition of done: what must be done to consider the task complete?

      There are two obvious solutions:

      1. We could check all document sizes when advancing the initial batch and return an error if any of the documents exceed the max length. This would ensure that nothing is inserted. This might be considered a breaking change.
      2. We could create a new error type that contains an WriteError that can be returned to the mongo layer. We can then include the results but remove the ids for the errors defined by the index of the write error, i.e. large documents.

      The exact Go version used, with patch level:

      go version go1.23.1 darwin/arm64

      The exact version of the Go driver used:

      1.17.1

      Describe how MongoDB is set up. Local vs Hosted, version, topology, load balanced, etc.

      Using MongoDB:          7.0.9
      Using Mongosh:          2.3.1
      

      The operating system and version (e.g. Windows 7, OSX 10.8, ...)

      OSX 14.5

      Security Vulnerabilities

      NA

            Assignee:
            Unassigned Unassigned
            Reporter:
            preston.vasquez@mongodb.com Preston Vasquez
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: