[SERVER-19372] Text query perf regression in commit d764e3e Created: 13/Jul/15  Updated: 06/Dec/22  Resolved: 05/Aug/19

Status: Closed
Project: Core Server
Component/s: Performance, Querying, Text Search
Affects Version/s: 3.1.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: David Daly Assignee: Backlog - Query Team (Inactive)
Resolution: Won't Fix Votes: 1
Labels: 32qa, mpreg, query-44-grooming
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-20613 Performance Regression on Mongo-perf ... Closed
Assigned Teams:
Query
Operating System: ALL
Steps To Reproduce:

Need mongo-perf – available here: https://github.com/mongodb/mongo-perf

Can run all the text tests with

python benchrun.py -f testcases/*.js --includeFilter text --out out -s <PATH TO SHELL> -t 8 

To run a single test, just put its name in place of text

python benchrun.py -f testcases/*.js --includeFilter Queries.Text.FindSingle --out out -s <PATH TO SHELL> -t 8 

Participants:

 Description   

Mongo-perf shows a regression on all of the text based query tests as of commit d764e3e.

Regression is visible here: https://evergreen.mongodb.com/task/performance_linux_wt_standalone_query_d764e3e6cf1d8a7d510df2f724282f7a053cecda_15_07_09_15_35_39
Click on the trend graph.



 Comments   
Comment by Craig Homa [ 05/Aug/19 ]

Closing as future perf work should be based on a broad view of system perf instead of targeting old reports of perf regressions.

Comment by David Daly [ 01/Oct/15 ]

To close the discussion here, after further review SERVER-20613 appears to be caused by the two commits listed above and not anything more. I just closed it as a dupe. However, the performance target overrides were committed against SERVER-20613. When this ticket is fixed, the targets will need to be updated appropriately.

Comment by J Rassi [ 22/Sep/15 ]

Ah, thanks for clarifying that there are multiple commits involved, I missed that. The two commits are unrelated, so I retract my suggestion that we close this ticket as a dup. 37% is indeed a concerning magnitude for a regression of any sort over a major release, but I stand by my earlier statement about 92eac3b5, and so I'm back to the opinion that we should treat these separately.

As far as this ticket goes, it agree that it merits further investigation and a fix, as no noticeable regression was expected by its introduction. David Daly, can you confirm that this regression should be reproducible by running mongo-perf text tests against d764e3e and its parent and comparing the results between the two? If so, I'll move this ticket into "Needs Triage" and triage it with Dave at our next triage meeting (within the next few weeks).

Our team is unfortunately pretty booked for most of the remainder of the 3.1.x cycle, so I wouldn't be surprised if a fix doesn't make it into the 3.2 release, which is unfortunate but reflects the reality of our current release schedule.

Comment by David Daly [ 22/Sep/15 ]

jason.rassi there appear to be at least three distinct regressions.

  1. d764e3e6cf
  2. 92eac3b5
  3. And something earlier (BF-1172) which I haven't isolated yet from before the mongo-perf tests were fully up in evergreen.

The aggregate is net performance loss of up to 37%. Is a 37% considered to be of a reasonable magnitude for these changes? I can update the targets and mark the BF as a duplicate if it is. Similarly, if those two commits are close enough in intent, I don't object to closing this ticket as a duplicate of SERVER-19936.

Comment by J Rassi [ 22/Sep/15 ]

Assuming that the regression described by this ticket and the build failure ticket can reasonably be determined to be introduced by 92eac3b5, I suggest we close them both as a duplicate of SERVER-19936, and leave that ticket in Planning Bucket A. This seems consistent with the discussion I had with Dan about this last month, which I summarized in a comment at SERVER-19936. Quoting here:

I briefly discussed with Dan.

Yes, this slowdown is expected and of a reasonable magnitude. We should do a performance pass on the new code to make up for some of the regression, but we do not think this should be scheduled for 3.1.x and are happy to ship with this feature performing as-is.

If there's new information available now that you think would change this decision then we can reconsider, but just based on a cursory scan of the tickets filed I am currently not much more concerned than I was a month ago.

mark.benvenuto, would you mind weighing in for a second opinion?

Comment by David Daly [ 22/Sep/15 ]

jason.rassi david.storch I don't think Adam is going to get to this one. What do we want to do with it? There appear to be 3 different text performance regressions (this, SERVER-19936, and some more caught in BF-1172). The sum total is pretty big.

Comment by David Daly [ 13/Jul/15 ]

Talked to david.storch and rassi@10gen.com about this. The commit is refactor. From discussion, adam.chelminski should investigate the cause of the performance regression. Based on that investigation, they will decide whether to fix or to leave as is.

adam.chelminski please take a look.

Generated at Thu Feb 08 03:50:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.