Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16626

wiredtiger group commit too slow for single-threaded workloads

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.8.0-rc3
    • Component/s: None
    • Labels:
      None
    • Fully Compatible
    • ALL
    • Hide

      Run iibench – https://github.com/tmcallaghan/iibench-mysql
      Change run.simple.bash from it with:
      export MAX_ROWS=100000000
      export NUM_DOCUMENTS_PER_INSERT=1
      export WRITE_CONCERN=FSYNC_SAFE

      Start mongod by: mongod --config /data/mysql/mongo.28/mongo.conf --storageEngine wiredTiger

      Using this mongo.conf. Repeating the test with journal compression enabled makes no difference.
      processManagement:
      fork: true
      systemLog:
      destination: file
      path: /data/mysql/mongo.28/log
      logAppend: true
      storage:
      dbPath: /data/mysql/mongo.28/data
      journal:
      enabled: true
      mmapv1:
      syncPeriodSecs: 60
      journal:
      commitIntervalMs: 100
      storage.wiredTiger.collectionConfig.blockCompressor: none
      storage.wiredTiger.engineConfig.journalCompressor: none

      Show
      Run iibench – https://github.com/tmcallaghan/iibench-mysql Change run.simple.bash from it with: export MAX_ROWS=100000000 export NUM_DOCUMENTS_PER_INSERT=1 export WRITE_CONCERN=FSYNC_SAFE Start mongod by: mongod --config /data/mysql/mongo.28/mongo.conf --storageEngine wiredTiger Using this mongo.conf. Repeating the test with journal compression enabled makes no difference. processManagement: fork: true systemLog: destination: file path: /data/mysql/mongo.28/log logAppend: true storage: dbPath: /data/mysql/mongo.28/data journal: enabled: true mmapv1: syncPeriodSecs: 60 journal: commitIntervalMs: 100 storage.wiredTiger.collectionConfig.blockCompressor: none storage.wiredTiger.engineConfig.journalCompressor: none

      I ran iibench (https://github.com/tmcallaghan/iibench-mysql) changed to insert 1 document per write and fsync enabled. My storage is fast (PCIe flash). With 10 threads I get more than 10,000 docs inserted per second. When I change to 1 thread I get ~20 docs inserted per second (or 1 per 50 milliseconds).

      So then I start to look at WiredTiger source for a 50 millisecond sleep and find this in db/storage/wiredtiger/wiredtiger_recovery_unit.cpp

                  // return true if happened
                  bool awaitCommit() {
                      boost::mutex::scoped_lock lk( mutex );
                      long long start = lastSyncTime;
                      numWaitingForSync.fetchAndAdd(1);
                      condvar.timed_wait(lk,boost::posix_time::milliseconds(50));
                      numWaitingForSync.fetchAndAdd(-1);
                      return lastSyncTime > start;
                  }
      

      A typical thread stack during the single threaded test is:
      pthread_cond_timedwait@@GLIBC_2.3.2,timed_wait<boost::unique_lock<boost::mutex>,timed_wait<boost::unique_lock<boost::mutex>,,awaitCommit,mongo::WiredTigerRecoveryUnit::awaitCommit,mongo::waitForWriteConcern,mongo::CmdGetLastError::run,mongo::_execCommand,mongo::Command::execCommand,mongo::_runCommands,runCommands,mongo::runQuery,receivedQuery,mongo::assembleResponse,mongo::MyMessageHandler::process,mongo::PortMessageServer::handleIncomingMsg

            Assignee:
            Unassigned Unassigned
            Reporter:
            mdcallag Mark Callaghan
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: