[SERVER-45488] detect_outliers doesn't appear to be running Created: 13/Dec/19  Updated: 29/Oct/23  Resolved: 13/Jan/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.3.3

Type: Bug Priority: Trivial - P5
Reporter: James O'Leary Assignee: Alexander Costas (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Backwards Compatibility: Fully Compatible
Sprint: DAG 2020-01-13, DAG 2020-01-27
Participants:
Story Points: 2

 Description   

detect_outliers should be running in system_perfl.yml.

But because it doesn't set -o verbose, we can't see if it runs or not:

 [2019/12/12 16:36:31.301] # detect outliers needs to run, so defer the post_run_check exit status to later
 [2019/12/12 16:36:31.301] echo $? > post_run_check.status
 [2019/12/12 16:36:31.302] Running command 'shell.exec' in "analyze" (step 4.3 of 4)
 [2019/12/12 16:36:31.305] Running command 'shell.exec' in "analyze" (step 4.4 of 4)
 [2019/12/12 16:36:31.306] filename=rejects.json
 [2019/12/12 16:36:31.306] if [ -s "$filename" ]; then
 [2019/12/12 16:36:31.306]   echo "Rejecting task due to the following outliers:"
 [2019/12/12 16:36:31.306]   cat "$filename"

See step 4.4 of 4.

Also the artifact doesn't contain a detect_outliers.log.

Finally, there have been no outliers saved in the outliers collection for some time:

MongoDB Enterprise PerformanceData-shard-0:PRIMARY> db.outliers.find({"project" : "sys-perf"}).sort({_id:-1}).limit(1)[0]._id.getTimestamp()
ISODate("2019-11-11T12:19:07Z")



 Comments   
Comment by Githook User [ 13/Jan/20 ]

Author:

{'name': 'Alexander Costas', 'email': 'alexander.costas@mongodb.com', 'username': 'ancostas'}

Message: SERVER-45488: detect_outliers doesn't appear to be running
Branch: master
https://github.com/mongodb/mongo/commit/92f2eb0263feb59adea80eb8115614623a2e213a

Comment by David Bradford (Inactive) [ 31/Dec/19 ]

I believe sys-perf patching has been disabled until after the new year to avoid overspending while lots of people are out of the office.

Comment by Alexander Costas (Inactive) [ 31/Dec/19 ]

brian.samek jim.oleary Can we no longer patch on sys-perf?

evergreen patch -u -p sys-perf -y -d TIG-2229
Unexpected reply from server (401 Unauthorized): 400 (Bad Request): patching is disabled

Comment by David Bradford (Inactive) [ 19/Dec/19 ]

We can look at pulling this in sooner. I have this as a low priority since I don't believe that anyone is using this data (and probably shouldn't be using this data, since we are making no guarantees about it right now, as is shown by this ticket.) 

Comment by April Schoffer [ 19/Dec/19 ]

david.bradford Is there any way we can plan this work sooner so we don't lose over a month of data?

Generated at Thu Feb 08 05:08:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.