[SERVER-29148] Increase run time for mongos tests in sys-perf Created: 11/May/17  Updated: 30/Oct/23  Resolved: 29/May/17

Status: Closed
Project: Core Server
Component/s: Performance
Affects Version/s: None
Fix Version/s: 3.2.15, 3.4.6, 3.5.8

Type: Task Priority: Major - P3
Reporter: Henrik Ingo (Inactive) Assignee: Henrik Ingo (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.4, v3.2
Sprint: Performance Kanban
Participants:

 Description   

It seems that mongos sys-perf tests are running a bit longer recently, and including the validate part, the mmapv1 task is at risk at hitting the 6 hour limit.

We should increase the exec_timeout_secs to 8 hours for that task. While doing that, please double check superficially that there isn't anything wrong why the task takes longer. (The assumption is that addition of fio and changing to EBS has made total runtime slower, even if the test results don't seem to have regressed.)



 Comments   
Comment by Githook User [ 20/Jun/17 ]

Author:

{u'username': u'henrikingo', u'name': u'Henrik Ingo', u'email': u'henrik.ingo@mongodb.com'}

Message: SERVER-29148 Increase exec_timeout_secs for sys-perf mongos_workloads_MMAPv1 to 9

...hours to leave some additional headroom

(cherry picked from commit f7dce408e025a4d7d807aeab9f4fa7cbe9bf5953)
Branch: v3.2
https://github.com/mongodb/mongo/commit/386d82060df58537bbdcb223f0ef54f97e7630ab

Comment by Githook User [ 20/Jun/17 ]

Author:

{u'username': u'henrikingo', u'name': u'Henrik Ingo', u'email': u'henrik.ingo@mongodb.com'}

Message: SERVER-29148 Increase run time for mongos tests in sys-perf

(cherry picked from commit 0ecf1da443f2cdc276e852217fbea8d4eb1c9df1)
Branch: v3.2
https://github.com/mongodb/mongo/commit/71d78971263ed816a37658186bb2e460166f5603

Comment by Githook User [ 20/Jun/17 ]

Author:

{u'username': u'henrikingo', u'name': u'Henrik Ingo', u'email': u'henrik.ingo@mongodb.com'}

Message: SERVER-29148 Increase exec_timeout_secs for sys-perf mongos_workloads_MMAPv1 to 9

...hours to leave some additional headroom

(cherry picked from commit f7dce408e025a4d7d807aeab9f4fa7cbe9bf5953)
Branch: v3.4
https://github.com/mongodb/mongo/commit/49a1983cd0f262dd9ded1b452014c91c9a623bc4

Comment by Githook User [ 20/Jun/17 ]

Author:

{u'username': u'henrikingo', u'name': u'Henrik Ingo', u'email': u'henrik.ingo@mongodb.com'}

Message: SERVER-29148 Increase run time for mongos tests in sys-perf

(cherry picked from commit 0ecf1da443f2cdc276e852217fbea8d4eb1c9df1)
Branch: v3.4
https://github.com/mongodb/mongo/commit/2e764884421c5e8e687bb37f0130839c3ed08fec

Comment by Githook User [ 29/May/17 ]

Author:

{u'username': u'henrikingo', u'name': u'Henrik Ingo', u'email': u'henrik.ingo@mongodb.com'}

Message: SERVER-29148 Increase exec_timeout_secs for sys-perf mongos_workloads_MMAPv1 to 9

...hours to leave some additional headroom
Branch: master
https://github.com/mongodb/mongo/commit/f7dce408e025a4d7d807aeab9f4fa7cbe9bf5953

Comment by Henrik Ingo (Inactive) [ 26/May/17 ]

Task now works, but run time was 7h45m. Re-opening this ticket to set time to 9 h instead.

Comment by Githook User [ 24/May/17 ]

Author:

{u'username': u'henrikingo', u'name': u'Henrik Ingo', u'email': u'henrik.ingo@mongodb.com'}

Message: SERVER-29148 Increase run time for mongos tests in sys-perf
Branch: master
https://github.com/mongodb/mongo/commit/0ecf1da443f2cdc276e852217fbea8d4eb1c9df1

Comment by Henrik Ingo (Inactive) [ 24/May/17 ]

kaloian.manassiev, others

It looks like the post-test validate_collections.js (and siblings in jstests/hooks) used to run in about 15 minutes, and after changing to readahead=32, it takes 30-40 minutes. It is run several times in the task, so combined this explains the increased run time.

Comment by Henrik Ingo (Inactive) [ 15/May/17 ]

kaloian.manassiev We will have to double check exactly that question when we actively work on this ticket, however based on a first glance the test results themselves haven't changed much. The increased run-time can come from activities outside the actual tests, most of which are also single threaded, so particularly sensitive:

  • inserts to populate a collection with data before a test
  • post-test checks in jstests/hooks (e.g. run_check_repl_dbhash.js)

If the increased run-time is due to issues like this, then the most likely reason is our recent migration to EBS instead of SSD, which increased latency of disk access. (But also made the latency more stable, which is what we like.)

Comment by Kaloian Manassiev [ 13/May/17 ]

Is this an indication of performance degradation? Is there a specific commit after which they have started running slower that you know of?

Generated at Thu Feb 08 04:20:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.