Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18908

Secondaries unable to keep up with primary under WiredTiger

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical - P2
    • Resolution: Fixed
    • 3.0.3, 3.0.4
    • 3.4.0-rc2
    • Performance, WiredTiger
    • Fully Compatible
    • ALL
    • Quint Iteration 7, QuInt A (10/12/15), QuInt C (11/23/15), Repl 2016-08-29, Repl 2016-09-19, Repl 2016-10-10, Repl 2016-10-31

    Description

      • hardware: 24 CPUs, 64 GB memory, SSD (all mongods and clients on same machine)
      • start 3-member replica set with following options: 

        mongod --oplogSize 50 --storageEngine wiredTiger --nojournal --replSet ...

        Note: repros with journal also, ran without journal to rule out that as a cause.

      • Simple small document insert workload: 5-16 threads (number doesn't matter to the repro) each inserting small documents {_id:..., x:0} in batches of 10k

      Replica lag grows unbounded as secondaries process ops at maybe 50-80% the rate of the primary.

      Some stats of note:

      primary

      secondary

      • op rate on secondary is maybe half that on primary
      • ops in flight (i.e. active queues) is much less even on the secondary, although that isn't reflected in the reported op rates
      • secondary is executing far more search near calls, about one per document, vs what appears to be about one every 100 documents on primary

      Will get stack traces.

      Attachments

        1. insert-3.0.4.patch
          3 kB
        2. insert-3.1.4.patch
          2 kB
        3. lag-02.html
          991 kB
        4. lag-03.html
          496 kB
        5. lag-04.html
          769 kB
        6. lagC-3.0.4.patch
          18 kB
        7. lagC-3.1.4.patch
          17 kB
        8. lagD-3.0.4.patch
          20 kB
        9. log.png
          log.png
          60 kB
        10. pri.png
          pri.png
          100 kB
        11. search_near.png
          search_near.png
          237 kB
        12. sec.png
          sec.png
          104 kB

        Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. SERVER-19122 Out-of-order _id index inserts on secondary contribute to replica lag under WiredTiger

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. WT-1989 Improvements to log slot freeing to improve thread scalability

          • Major - P3 - Major loss of function.
          • Closed

          Task - A task that needs to be done. SERVER-18983 Process oplog inserts, and applying, on the secondary in parallel

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-15344 Make replWriter thread pool size tunable

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-18982 Apply replicated inserts as inserts

          • Major - P3 - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SERVER-21444 WiredTiger replica set member is getting behind while syncing the oplog during the initialSync

          • Critical - P2 - Crashes, loss of data, severe memory leak.
          • Closed
          related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-19668 Waiting time for write concern increase as time goes by

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SERVER-21858 A high throughput update workload in a replica set can cause starvation of secondary reads

          • Major - P3 - Major loss of function.
          • Closed

          Activity

            People

              mathias@mongodb.com Mathias Stearn
              bruce.lucas@mongodb.com Bruce Lucas (Inactive)
              Votes:
              11 Vote for this issue
              Watchers:
              53 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: