Add QoS features to MongoDB

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Won't Do
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Usability
    • None
    • Service Arch
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      I am currently writing an application that generates sustained, heavy writes for about one hour at >200Mbit/s. This application is regularly causing performance issues on the mongodb side.

      It currently runs relatively fine on our atlas production setup using write-concern: majority and a single insertation thread, but it is causing very high loads (>90% disk utilization) on our test setup. This is in would be desired (maxing out the server's resources is good because we get more performance), but we had a db crash in the past (https://jira.mongodb.org/browse/MMSSUPPORT-14543 caused probably by a performance difference between the primary and replication – https://jira.mongodb.org/browse/SERVER-24242) and we are still not sure this could not happen again.

      In addition, using write-concern: majority causes a significant latency wich seems to be slowing down our application by a lot (now taking 3h instead of 1h since I activated this).

      It would be very useful if there was a way to explicitly mark operations as high-impact/best-effort/background jobs. Mongodb should make sure that all operations that are not marked in such a way are given priority before the high-impact tasks. Mongodb should also make sure that high-impact operations can never overtax the server; these operations should either be rejected or blocked until enough resources are available to process them.

      (Blocking would be very useful because then on my side I could write a scheduler that reduces the number of insertation threads or throttles them when many jobs are rejected).

      This way it would be possible to operate high-load jobs at maximum efficiency without stealing resources from routine operations.

              Assignee:
              [DO NOT USE] Backlog - Service Architecture
              Reporter:
              Karolin Varner
              Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

                Created:
                Updated:
                Resolved: