Uploaded image for project: 'C Driver'
  1. C Driver
  2. CDRIVER-1213

Pre-split bulk ops every 1000 documents

    • Type: Icon: New Feature New Feature
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 1.4.0
    • Affects Version/s: None
    • Component/s: Bulk API, Performance
    • Labels:
      None

      The TestSmallDocBulkInsert task of the driver benchmarks reveals an inefficiency: as you call mongoc_bulk_operation_insert all docs are appended to the same buffer. In mongoc_bulk_operation_execute, those catenated documents are then split into batches of 1000, the batches have to be copied into "insert" command documents before they're sent.

      The simplest fix is to pre-split batches: when the current batch has 1000 documents, mongoc_bulk_operation_insert should start a new one.

      Same for updates and deletes.

      A bulk op can be constructed before the driver has selected a server to send it to, or before the driver has connected at all, so the server's maxWriteBatchSize isn't known. Assume 1000 for now.

            Assignee:
            jesse@mongodb.com A. Jesse Jiryu Davis
            Reporter:
            jesse@mongodb.com A. Jesse Jiryu Davis
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: