Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16575

intermittent slow inserts with WiredTiger b-tree

    XMLWordPrintableJSON

Details

    • Storage Execution
    • ALL
    • Hide

      Run iibench (https://github.com/tmcallaghan/iibench-mongodb) with 2 things changed in run.simple.bash. This uses 10 client threads to load.
      export MAX_ROWS=200000000
      export LENGTH_CHAR_FIELDS=2000

      This problem reproduces for this configuration:
      bin.tcm/mongod --config /data/mysql/mongo.28/mongo.conf --storageEngine wiredTiger --wiredTigerEngineConfig="cache_size=32G,direct_io=[data]"

      And I use this mongo.conf:
      processManagement:
      fork: true
      systemLog:
      destination: file
      path: /data/mysql/mongo.28/log
      logAppend: true
      storage:
      dbPath: /data/mysql/mongo.28/data
      journal:
      enabled: true
      mmapv1:
      syncPeriodSecs: 60
      journal:
      commitIntervalMs: 100
      I wasn't able to reproduce it for this configuration:
      bin.tcm/mongod --config /data/mysql/mongo.28/mongo.conf --storageEngine wiredTiger

      Show
      Run iibench ( https://github.com/tmcallaghan/iibench-mongodb ) with 2 things changed in run.simple.bash. This uses 10 client threads to load. export MAX_ROWS=200000000 export LENGTH_CHAR_FIELDS=2000 This problem reproduces for this configuration: bin.tcm/mongod --config /data/mysql/mongo.28/mongo.conf --storageEngine wiredTiger --wiredTigerEngineConfig="cache_size=32G,direct_io= [data] " And I use this mongo.conf: processManagement: fork: true systemLog: destination: file path: /data/mysql/mongo.28/log logAppend: true storage: dbPath: /data/mysql/mongo.28/data journal: enabled: true mmapv1: syncPeriodSecs: 60 journal: commitIntervalMs: 100 I wasn't able to reproduce it for this configuration: bin.tcm/mongod --config /data/mysql/mongo.28/mongo.conf --storageEngine wiredTiger

    Description

      I get intermittent slow inserts with the WiredTiger b-tree in 2.8.0rc2. These are the top-10 worst ones from a test in progress. The numbers are milliseconds, so there are many inserts that take more than 100 seconds:
      119597
      118610
      118465
      117977
      117437
      116611
      116498
      116068
      113909
      113342

      This doesn't reproduce with the WiredTiger LSM. I wonder if a slow checkpoint is the problem but I have not done the work to trigger collection of thread stacks when there is a stall, in part because my client code and the error log is printed when the stall ends.

      I will repeat the test with PMP running to get thread stacks every 10 seconds to see if I get stack when the problem occurs.

      Attachments

        1. jmongoiibench.class
          11 kB
        2. jmongoiibench.java
          35 kB
        3. jmongoiibench$MyQuery.class
          4 kB
        4. jmongoiibench$MyReporter.class
          4 kB
        5. jmongoiibench$MyWriter.class
          4 kB
        6. performance_output.png
          performance_output.png
          204 kB
        7. run.simple.bash.L10.Q1
          4 kB

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-17407 Significant performance degradation between SA mongod to replica set on insert only workload

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-16665 WiredTiger b-tree uses too much space in journal directory

          • Major - P3 - Major loss of function.
          • Closed
          related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-17157 Seeing pauses in YCSB performance workload with WiredTiger

          • Critical - P2 - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              backlog-server-execution Backlog - Storage Execution Team
              mdcallag Mark Callaghan
              Votes:
              3 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: