[SERVER-32298] Add a background system task to periodically copy remote statistics in evergreen.yml Created: 13/Dec/17  Updated: 30/Oct/23  Resolved: 20/Dec/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.6.3, 3.7.1

Type: Task Priority: Major - P3
Reporter: Jonathan Abrahams Assignee: Jonathan Abrahams
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Problem/Incident
Related
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.6
Sprint: TIG 2018-1-1
Participants:
Linked BF Score: 0

 Description   

For remotely managed hosts, which do not have an Evergreen agent (powercycle task), the post phase copies system and process statistics. In the case of a failure to reach this host these statistics are lost. We can periodically copy them (every 30 seconds), so the information is not completely lost, by doing it in a background system task, "copy ec2 statistics".



 Comments   
Comment by Githook User [ 02/Feb/18 ]

Author:

{'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams', 'username': 'hptabster'}

Message: SERVER-32298 Add a background system task to periodically copy remote statistics in evergreen.yml

(cherry picked from commit 9d6383c2b9f0c6af06d7181e6cff9152d92ba858)
Branch: v3.6
https://github.com/mongodb/mongo/commit/8f26f02842637625ac23a1e9fb2518c12c5375ed

Comment by Githook User [ 20/Dec/17 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-32298 Add a background system task to periodically copy remote statistics in evergreen.yml
Branch: master
https://github.com/mongodb/mongo/commit/9d6383c2b9f0c6af06d7181e6cff9152d92ba858

Comment by Max Hirschhorn [ 20/Dec/17 ]

I reverted the changes from 901760c since it was causing the burn_in_tests Evergreen task to fail. This is because burn_in_tests.py attempts to parse etc/evergreen.yml as valid YAML. It is invalid to attempt to use an alias prior to it being defined as an anchor; however, the gopkg.in/yaml.v2 package used by Evergreen deviates from the YAML specification. CC sam.kleinman

An alias node is denoted by the “*” indicator. The alias refers to the most recent preceding node having the same anchor. It is an error for an alias node to use an anchor that does not previously occur in the document. It is not an error to specify an anchor that is not used by any alias node.

http://www.yaml.org/spec/1.2/spec.html#id2786196

[2017/12/19 21:03:44.718] $python buildscripts/burn_in_tests.py --branch=master --buildVariant=enterprise-rhel-62-64-bit --testListOutfile=jstests/new_tests.json --noExec $burn_in_args
[2017/12/19 21:03:45.235] Traceback (most recent call last):
[2017/12/19 21:03:45.236]   File "buildscripts/burn_in_tests.py", line 418, in <module>
[2017/12/19 21:03:45.236]     main()
[2017/12/19 21:03:45.236]   File "buildscripts/burn_in_tests.py", line 366, in main
[2017/12/19 21:03:45.236]     evergreen_conf = evergreen.EvergreenProjectConfig(values.evergreen_file)
[2017/12/19 21:03:45.236]   File "/data/mci/d7c78ddfd9f582185efcc68e71f35dc7/burn_in_tests_clonedir/buildscripts/ciconfig/evergreen.py", line 19, in __init__
[2017/12/19 21:03:45.236]     self._conf = yaml.load(fstream)
[2017/12/19 21:03:45.236]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/__init__.py", line 71, in load
[2017/12/19 21:03:45.237]     return loader.get_single_data()
[2017/12/19 21:03:45.237]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/constructor.py", line 37, in get_single_data
[2017/12/19 21:03:45.238]     node = self.get_single_node()
[2017/12/19 21:03:45.238]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node
[2017/12/19 21:03:45.239]     document = self.compose_document()
[2017/12/19 21:03:45.239]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/composer.py", line 55, in compose_document
[2017/12/19 21:03:45.239]     node = self.compose_node(None, None)
[2017/12/19 21:03:45.239]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
[2017/12/19 21:03:45.239]     node = self.compose_mapping_node(anchor)
[2017/12/19 21:03:45.239]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/composer.py", line 133, in compose_mapping_node
[2017/12/19 21:03:45.239]     item_value = self.compose_node(node, item_key)
[2017/12/19 21:03:45.239]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
[2017/12/19 21:03:45.239]     node = self.compose_mapping_node(anchor)
[2017/12/19 21:03:45.239]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/composer.py", line 133, in compose_mapping_node
[2017/12/19 21:03:45.239]     item_value = self.compose_node(node, item_key)
[2017/12/19 21:03:45.239]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/composer.py", line 82, in compose_node
[2017/12/19 21:03:45.239]     node = self.compose_sequence_node(anchor)
[2017/12/19 21:03:45.239]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/composer.py", line 111, in compose_sequence_node
[2017/12/19 21:03:45.239]     node.value.append(self.compose_node(node, index))
[2017/12/19 21:03:45.239]   File "/opt/mongodbtoolchain/v2/lib/python2.7/site-packages/yaml/composer.py", line 69, in compose_node
[2017/12/19 21:03:45.240]     % anchor.encode('utf-8'), event.start_mark)
[2017/12/19 21:03:45.240] yaml.composer.ComposerError: found undefined alias 'copy_ec2_monitor_files'
[2017/12/19 21:03:45.263]   in "etc/evergreen.yml", line 1412, column 7
[2017/12/19 21:03:45.263] Command failed: script finished with error: exit status 1

Comment by Githook User [ 20/Dec/17 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: Revert "SERVER-32298 Add a background system task to periodically copy remote statistics in evergreen.yml"

This reverts commit 901760c683dbaf4c1581bf7216ca80756adcc581.
Branch: master
https://github.com/mongodb/mongo/commit/3865b0b5b9b981e40a4bc6815dd9ef3bcf2bd2fe

Comment by Githook User [ 19/Dec/17 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-32298 Add a background system task to periodically copy remote statistics in evergreen.yml
Branch: master
https://github.com/mongodb/mongo/commit/901760c683dbaf4c1581bf7216ca80756adcc581

Generated at Thu Feb 08 04:29:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.