ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Networking & Observability
    • ALL
    • Hide
      systemLog:
        verbosity: 0
        destination: file
        path: /data3/yyz_test/mongodb_pipeline/replica_set/37003/logs/mongod.log
        logAppend: true
        logRotate: renameprocessManagement:
        fork: true
        pidFilePath: /data3/yyz_test/mongodb_pipeline/replica_set/37003/mongod.pidnet:
        port: 37003
        bindIpAll: true
        maxIncomingConnections: 10000
        ipv6: true
      #  compression: {}
        compression:
          compressors: zstdsetParameter:
        disabledSecureAllocatorDomains: '*'
        tcmallocAggressiveMemoryDecommit: 1
        cursorTimeoutMillis: 600000
        maxTransactionLockRequestTimeoutMillis: 5
        internalQueryMaxBlockingSortMemoryUsageBytes: 33554432storage:
        dbPath: /data4/yyz_test/mongodb_pipeline/replica_set/37003/data
        directoryPerDB: true
        syncPeriodSecs: 60
        wiredTiger:
          engineConfig:
            cacheSizeGB: 24
            journalCompressor: none
          evictionConfig: {}
          collectionConfig:
            blockCompressor: snappy
            
      replication:
        oplogSizeMB: 25600
        replSetName: rs0operationProfiling:
        mode: "off"
        slowOpThresholdMs: 100security:
        keyFile: /data3/yyz_test/mongodb_pipeline/replica_set/keyfile
        authorization: enabled
      root@:/data3/yyz_test/mongodb_pipeline# cat /data3/yyz_test/mongodb_pipeline/replica_set/37001/mongod.conf
      systemLog:
        verbosity: 0
        destination: file
        path: /data3/yyz_test/mongodb_pipeline/replica_set/37001/logs/mongod.log
        logAppend: true
        logRotate: renameprocessManagement:
        fork: true
        pidFilePath: /data3/yyz_test/mongodb_pipeline/replica_set/37001/mongod.pidnet:
        port: 37001
        bindIpAll: true
        maxIncomingConnections: 10000
        ipv6: true
      #  compression: {}
        compression:
          compressors: zstdsetParameter:
        disabledSecureAllocatorDomains: '*'
        tcmallocAggressiveMemoryDecommit: 1
        cursorTimeoutMillis: 600000
        maxTransactionLockRequestTimeoutMillis: 5
        internalQueryMaxBlockingSortMemoryUsageBytes: 33554432
        internalReplPipelineCompressionThreadPoolSize: 10storage:
        dbPath: /data2/yyz_test/mongodb_pipeline/replica_set/37001/data
        directoryPerDB: true
        syncPeriodSecs: 60
        wiredTiger:
          engineConfig:
            cacheSizeGB: 24
            journalCompressor: none
          evictionConfig: {}
          collectionConfig:
            blockCompressor: snappy
            
      replication:
        oplogSizeMB: 25600
        replSetName: rs0operationProfiling:
        mode: "off"
        slowOpThresholdMs: 100security:
        keyFile: /data3/yyz_test/mongodb_pipeline/replica_set/keyfile
        authorization: enabled 

      1 primary + 2 secondary:

      ./bin/ycsb.sh load mongodb -s -P workloads/workloada -p mongodb.url='mongodb://root:xxxxx@xxxxx:37001,xxxxx:37002,xxxxx:37003/admin?maxPoolSize=500&minPoolSize=500&waitQueueTimeoutMS=120000' -p fieldcount=5000 -threads 100 -p recordcount=100000 -p table=test21 -threads 1000 

       

      On a replica set with network (wire) compression enabled, replication sync-source responses (getMore on local.oplog.rs) are compressed inline on the connection / session-workflow thread. For workloads dominated by large documents this makes a single replication stream CPU-bound on one core: the connection thread serializes read oplog → build response → compress → send, and compression dominates. As a result, replica-set write throughput is capped well below what the storage engine and disk can sustain, even on very fast NVMe.

      This ticket proposes an asynchronous, order-preserving compression pipeline: response compression is offloaded from the connection thread to a small shared worker pool, while the connection thread stays the sole sender and preserves strict wire ordering. This lets the sync-source thread keep producing exhaust getMore responses while previous batches are compressed in parallel.

      Measured with a YCSB large-document write load on an 8c/24G node (snappy) backed by ~1M-IOPS NVMe, the pipeline lifts insert throughput from ~990 ops/s to ~1377 ops/s (about +39%) and removes the single-connection compression ceiling. With the more CPU-intensive zstd compressor on a 12c/32G node the gain is even larger (~3.85×, see the zstd section).

      Show
      systemLog:   verbosity: 0   destination: file   path: /data3/yyz_test/mongodb_pipeline/replica_set/37003/logs/mongod.log   logAppend: true   logRotate: renameprocessManagement:   fork: true   pidFilePath: /data3/yyz_test/mongodb_pipeline/replica_set/37003/mongod.pidnet:   port: 37003   bindIpAll: true   maxIncomingConnections: 10000   ipv6: true #  compression: {}   compression:     compressors: zstdsetParameter:   disabledSecureAllocatorDomains: '*'   tcmallocAggressiveMemoryDecommit: 1   cursorTimeoutMillis: 600000   maxTransactionLockRequestTimeoutMillis: 5   internalQueryMaxBlockingSortMemoryUsageBytes: 33554432storage:   dbPath: /data4/yyz_test/mongodb_pipeline/replica_set/37003/data   directoryPerDB: true   syncPeriodSecs: 60   wiredTiger:     engineConfig:       cacheSizeGB: 24       journalCompressor: none     evictionConfig: {}     collectionConfig:       blockCompressor: snappy        replication:   oplogSizeMB: 25600   replSetName: rs0operationProfiling:   mode: "off"   slowOpThresholdMs: 100security:   keyFile: /data3/yyz_test/mongodb_pipeline/replica_set/keyfile   authorization: enabled root@:/data3/yyz_test/mongodb_pipeline# cat /data3/yyz_test/mongodb_pipeline/replica_set/37001/mongod.conf systemLog:   verbosity: 0   destination: file   path: /data3/yyz_test/mongodb_pipeline/replica_set/37001/logs/mongod.log   logAppend: true   logRotate: renameprocessManagement:   fork: true   pidFilePath: /data3/yyz_test/mongodb_pipeline/replica_set/37001/mongod.pidnet:   port: 37001   bindIpAll: true   maxIncomingConnections: 10000   ipv6: true #  compression: {}   compression:     compressors: zstdsetParameter:   disabledSecureAllocatorDomains: '*'   tcmallocAggressiveMemoryDecommit: 1   cursorTimeoutMillis: 600000   maxTransactionLockRequestTimeoutMillis: 5   internalQueryMaxBlockingSortMemoryUsageBytes: 33554432   internalReplPipelineCompressionThreadPoolSize: 10storage:   dbPath: /data2/yyz_test/mongodb_pipeline/replica_set/37001/data   directoryPerDB: true   syncPeriodSecs: 60   wiredTiger:     engineConfig:       cacheSizeGB: 24       journalCompressor: none     evictionConfig: {}     collectionConfig:       blockCompressor: snappy        replication:   oplogSizeMB: 25600   replSetName: rs0operationProfiling:   mode: "off"   slowOpThresholdMs: 100security:   keyFile: /data3/yyz_test/mongodb_pipeline/replica_set/keyfile   authorization: enabled 1 primary + 2 secondary: ./bin/ycsb.sh load mongodb -s -P workloads/workloada -p mongodb.url= 'mongodb: //root:xxxxx@xxxxx:37001,xxxxx:37002,xxxxx:37003/admin?maxPoolSize=500&minPoolSize=500&waitQueueTimeoutMS=120000' -p fieldcount=5000 -threads 100 -p recordcount=100000 -p table=test21 -threads 1000   On a replica set with network (wire) compression enabled, replication sync-source responses ( getMore  on  local.oplog.rs ) are compressed  inline on the connection / session-workflow thread . For workloads dominated by large documents this makes a single replication stream  CPU-bound on one core : the connection thread serializes  read oplog → build response → compress → send , and compression dominates. As a result, replica-set write throughput is capped well below what the storage engine and disk can sustain, even on very fast NVMe. This ticket proposes an  asynchronous, order-preserving compression pipeline : response compression is offloaded from the connection thread to a small shared worker pool, while the connection thread stays the sole sender and preserves strict wire ordering. This lets the sync-source thread keep producing exhaust  getMore  responses while previous batches are compressed in parallel. Measured with a YCSB large-document write load on an 8c/24G node (snappy) backed by ~1M-IOPS NVMe, the pipeline lifts insert throughput from  ~990 ops/s to ~1377 ops/s (about +39%)  and removes the single-connection compression ceiling. With the more CPU-intensive zstd compressor on a 12c/32G node the gain is even larger (~3.85×, see the zstd section).
    • N&O Prioritized List
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      1. image-2026-07-31-20-02-32-910.png
        179 kB
        y yz (EXT)
      2. image-2026-07-31-20-02-43-464.png
        157 kB
        y yz (EXT)
      3. image-2026-07-31-20-03-15-299.png
        203 kB
        y yz (EXT)
      4. image-2026-07-31-20-03-15-383.png
        203 kB
        y yz (EXT)
      5. image-2026-07-31-20-03-26-005.png
        184 kB
        y yz (EXT)
      6. image-2026-07-31-20-03-36-085.png
        252 kB
        y yz (EXT)
      7. image-2026-07-31-20-04-07-763.png
        125 kB
        y yz (EXT)
      8. image-2026-07-31-20-04-20-233.png
        307 kB
        y yz (EXT)
      9. image-2026-07-31-20-04-31-754.png
        252 kB
        y yz (EXT)
      10. image-2026-07-31-20-04-40-488.png
        240 kB
        y yz (EXT)
      11. image-2026-07-31-20-09-54-135.png
        159 kB
        y yz (EXT)
      12. image-2026-07-31-20-10-03-062.png
        149 kB
        y yz (EXT)
      13. image-2026-07-31-20-10-11-772.png
        186 kB
        y yz (EXT)
      14. image-2026-07-31-20-10-26-239.png
        300 kB
        y yz (EXT)
      15. image-2026-07-31-20-10-42-262.png
        296 kB
        y yz (EXT)
      16. image-2026-07-31-20-11-00-684.png
        261 kB
        y yz (EXT)
      17. image-2026-07-31-20-11-08-831.png
        152 kB
        y yz (EXT)
      18. image-2026-07-31-20-11-18-800.png
        180 kB
        y yz (EXT)
      19. image-2026-07-31-20-12-09-537.png
        377 kB
        y yz (EXT)
      20. image-2026-07-31-20-12-34-185.png
        359 kB
        y yz (EXT)
      21. image-2026-07-31-20-15-26-366.png
        465 kB
        y yz (EXT)
      22. MongoDB_Large_Document_Performance_AI_Gaming.pdf
        50 kB
        y yz (EXT)

          Assignee:
          Unassigned
          Reporter:
          y yz (EXT)
          Votes:
          0 Vote for this issue
          Watchers:
          6 Start watching this issue

            Created:
            Updated: