Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16385

significant regression in $push performance in 2.6.5 compared to 2.4.3

    • Query Execution

      I have been doing some benchmarking ahead of the migration of our live system from 2.4.3 to 2.6.5. The first item for concern that my testing has highlighted is that there appears to have been a significant regression in performance in Mongo's ability to cope with multiple pushes of subdocuments.
      My test scenario is as follows:
      I have two identically configured servers in the same availability zone in AWS (of instance type m1.large). One is running 2.4.3 and the other is 2.6.5.
      I have another larger box (hi1.4xlarge) in the same availability zone from which I execute the test first on one database, then the other.
      The test involves making 100k updates with a push of a single small subdocument to the same collection and measuring the time it takes when pushing 20, 100, 1000, 5000, 10000 subdocs per doc. The collection has several indices including a unique index on a UUID field.
      The test proceeds as follows:
      empty the collection
      generate appropriate number of UUIDs across which to spread 100k subdocs i.e. 100k / 20 = 5000 for the first test
      make 100k updates (with upsert=True) as fast as possible and log time taken / database stats

      From my scatter plot of results, it seems that the test runs fastest with 100 subdocs per doc (presumably because of the cost of updating the indices on the upsert of the larger number of documents dominates the time taken for the push with 20 subdocs per doc). Also it is clear that performance in 2.6.5 has been improved at this end of the scale.
      However, as the number of pushes per document is increased, the performance of 2.6.5 degrades much faster than 2.4.3. I tried both Mongo versions with and without UsePowerOf2Sizes but found the effect of this parameter to be negligible (which was surprising having read the documentation on this feature). Are there any other parameters that affect the allocation of space that could be negatively affecting the performance of 2.6.5 in this test?

      I attach scatter plot of timings, server stats from 2.4.3 and 2.6.5 when pushing 10k subdocs per doc and the python code used for testing (minus all the data logging stuff).

        1. 243noPow2.svg
          240 kB
          John Greenall
        2. 243usePow2.svg
          230 kB
          John Greenall
        3. 265usePow2.svg
          313 kB
          John Greenall
        4. mongoSubdocumentPushJira.py
          3 kB
          John Greenall
        5. noPow265.svg
          301 kB
          John Greenall
        6. testTimings.svg
          47 kB
          John Greenall

            Assignee:
            backlog-query-execution [DO NOT USE] Backlog - Query Execution
            Reporter:
            johng John Greenall
            Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: