[SERVER-39859] Use the Evergreen REST v2 API to download -latest tarballs in setup multiversion mongodb Created: 27/Feb/19  Updated: 29/Oct/23  Resolved: 08/Dec/20

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.9.0

Type: Improvement Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Mikhail Shchatko
Resolution: Fixed Votes: 0
Labels: neweng, tig-multiversion, tig-qwin-eligible
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-53047 Add powercycle smoke test Closed
Gantt Dependency
has to be done before SERVER-49869 Support downloading debug symbols in ... Closed
has to be done after SERVER-51776 Map setup_multiversion_mongodb.py arg... Closed
has to be done after SERVER-52534 Auth to the Evergreen API and S3 for ... Closed
has to be done after SERVER-52535 Use evergreen and github API to get t... Closed
Related
related to SERVER-47365 Master (v4.5) branch's last-stable is... Closed
is related to SERVER-35972 Run push tasks more often Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.4
Sprint: STM 2020-11-30, STM 2020-12-14
Participants:
Linked BF Score: 15
Story Points: 2

 Description   

There are caching policies for fastdl.mongodb.org and downloads.mongodb.com that cause --useLatest to download day-old or possibly month-old binaries even after the push task has succeeded on the older branches. Changing to use the /tasks/<task_id> endpoint (for example) would make it possible to download the files directly from S3.

{
  "name": "Binaries",
  "url": "https://s3.amazonaws.com/mciuploads/mongodb-mongo-v4.0/enterprise-rhel-62-64-bit/f35415816415b9ff93bb963690ac4f8c9f6bc453/binaries/mongo-mongodb_mongo_v4.0_enterprise_rhel_62_64_bit_f35415816415b9ff93bb963690ac4f8c9f6bc453_19_02_26_21_26_34.tgz",
  "visibility": "",
  "ignore_for_fetch": false
},

An unfortunate consequence of using Evergreen to download binaries from older branches is that we would tie build variants across different mongodb-mongo-* Evergreen projects together when we haven't necessarily been consistent about naming. This means it won't always be possible to download the binaries from the build variant with the same name as the one that exists in mongodb-mongo-master and we may need to keep an explicit mapping in etc/evergreen.yml.

We must also make a decision about whether we should depend on the push task succeeding, or only depend on compile, jsCore, dbtest, and replica_sets_jscore_passthrough succeeding as SERVER-35972 has changed for the 4.0 and master branches.



 Comments   
Comment by Githook User [ 08/Dec/20 ]

Author:

{'name': 'Mikhail Shchatko', 'email': 'mikhail.shchatko@mongodb.com'}

Message: SERVER-39859 Fix burn_in_tags_gen
Branch: master
https://github.com/mongodb/mongo/commit/534e48d46ed745ad3111e7d26f5ef70aeb51ff79

Comment by Githook User [ 08/Dec/20 ]

Author:

{'name': 'Mikhail Shchatko', 'email': 'mikhail.shchatko@mongodb.com'}

Message: SERVER-39859 Use the Evergreen REST v2 API to download -latest tarballs in setup multiversion mongodb
Branch: master
https://github.com/mongodb/mongo/commit/7fb2d5929683c61462b10550a0dd69d23b951954

Comment by Brooke Miller [ 02/Nov/20 ]

As we discussed in triaging, given SERVER-39859 and SERVER-52534, this ticket will be to confirm that this all works together by

  • deleting the current multiversion_setup.py script
  • use the new script that will be part of resmoke
  • update evergreen.yml to use the new script 
Comment by Mikhail Shchatko [ 21/Oct/20 ]

Picking out `map the current args to evergreen build variants` part into SERVER-51776

Comment by Robert Guo (Inactive) [ 15/Oct/20 ]

We're going to break this ticket up into multiple pieces.

1. auth to the Evergreen API and S3 for both local and Evergreen machines
2. map the current args to evergreen build variants
3. decide on if we want to use the API to get the bin version or use a fixed path for each release and for latest.
4. rewrite the multiversion client to use Evergreen's URLs instead of the public download links

Comment by Max Hirschhorn [ 29/Apr/20 ]

Currently, if one of the older binaries crashes and leaves a core dump, we have no way of using it, since the binary and debug symbols are not saved.

This came up with Blake and Marcos today when they were looking at a multiversion failure. I realized because the -latest tarballs come directly from the mongodb-mongo-v* Evergreen projects (i.e. the binaries aren't rebuilt from scratch like we do for releases) that it is very possible to get the corresponding binaries and debug symbols provided the logs are still available.

1. Search for the start-up message which says "git version" or "gitVersion".

[js_test:addshard5] 2020-04-29T14:05:22.089+0000 d20771| 2020-04-29T14:05:22.089+00:00 I  CONTROL  23403   [initandlisten] "Build Info","attr":{"buildInfo":{"version":"4.4.0-rc3-11-ge33cb0c","gitVersion":"e33cb0c81df6901645788f1df484c2d492c08b1f","openSSLVersion":"OpenSSL 1.0.1e-fips 11 Feb 2013","modules":["enterprise"],"allocator":"tcmalloc","environment":{"distmod":"rhel62","distarch":"x86_64","target_arch":"x86_64"}}}
...
[js_test:addshard5] 2020-04-29T14:05:32.301+0000 d20771| 2020-04-29T14:05:32.301+00:00 I  CONTROL  31431   [ConfigServerCatalogCacheLoader-0] "BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"7F9E248ABFE1","b":"7F9E21ADC000","o":"2DCFFE1","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.596","s+":"1E1"},{"a":"7F9E248AD6D9","b":"7F9E21ADC000","o":"2DD16D9","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"7F9E24898B11","b":"7F9E21ADC000","o":"2DBCB11","s":"_ZN5mongo11DBException13traceIfNeededERKS0_","s+":"B1"},{"a":"7F9E24828512","b":"7F9E21ADC000","o":"2D4C512","s":"_ZN5mongo11DBExceptionC2ERKNS_6StatusE","s+":"32"},{"a":"7F9E22A1C2DC","b":"7F9E21ADC000","o":"F402DC","s":"_ZN5mongo13error_details23throwExceptionForStatusERKNS_6StatusE","s+":"177C"},{"a":"7F9E22A2F389","b":"7F9E21ADC000","o":"F53389","s":"_ZN5mongo21uassertedWithLocationERKNS_6StatusEPKcj","s+":"27C"},{"a":"7F9E227BF57E","b":"7F9E21ADC000","o":"CE357E","s":"_ZN5mongo12_GLOBAL__N_132getPersistedMetadataSinceVersionEPNS_16OperationContextERKNS_15NamespaceStringENS_12ChunkVersionE.cold.1253","s+":"14"},{"a":"7F9E2311A78C","b":"7F9E21ADC000","o":"163E78C","s":"_ZN5mongo29ShardServerCatalogCacheLoader18_getLoaderMetadataEPNS_16OperationContextERKNS_15NamespaceStringERKNS_12ChunkVersionEx","s+":"BC"},{"a":"7F9E2311ED88","b":"7F9E21ADC000","o":"1642D88","s":"_ZNSt17_Function_handlerIFvPN5mongo16OperationContextENS0_10StatusWithINS0_18CatalogCacheLoader26CollectionAndChangedChunksEEEEZNS0_29ShardServerCatalogCacheLoader30_schedulePrimaryGetChunksSinceES2_RKNS0_15NamespaceStringERKNS0_12ChunkVersionExSt8functionIS7_ESt10shared_ptrINS0_12NotificationIvEEEEUlT_T0_E1_E9_M_invokeERKSt9_Any_dataOS2_OS6_","s+":"378"},{"a":"7F9E233B7789","b":"7F9E21ADC000","o":"18DB789","s":"_ZZN5mongo30ConfigServerCatalogCacheLoader14getChunksSinceERKNS_15NamespaceStringENS_12ChunkVersionESt8functionIFvPNS_16OperationContextENS_10StatusWithINS_18CatalogCacheLoader26CollectionAndChangedChunksEEEEEENKUlT_E_clINS_6StatusEEEDaSE_","s+":"1349"},{"a":"7F9E233B8927","b":"7F9E21ADC000","o":"18DC927","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_30ConfigServerCatalogCacheLoader14getChunksSinceERKNS_15NamespaceStringENS_12ChunkVersionESt8functionIFvPNS_16OperationContextENS_10StatusWithINS_18CatalogCacheLoader26CollectionAndChangedChunksEEEEEEUlT_E_EEDaOSJ_EN12SpecificImpl4callEOS1_","s+":"37"},{"a":"7F9E243DE5AB","b":"7F9E21ADC000","o":"29025AB","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockINS_12latch_detail5LatchEE","s+":"14B"},{"a":"7F9E243E0E01","b":"7F9E21ADC000","o":"2904E01","s":"_ZN5mongo10ThreadPool13_consumeTasksEv","s+":"81"},{"a":"7F9E243E1C3A","b":"7F9E21ADC000","o":"2905C3A","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE","s+":"FA"},{"a":"7F9E243E1F80","b":"7F9E21ADC000","o":"2905F80","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_10ThreadPool25_startWorkerThread_inlockEvEUlvE2_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"60"},{"a":"7F9E24A5AF0F","b":"7F9E21ADC000","o":"2F7EF0F","s":"execute_native_thread_routine","s+":"F"},{"a":"7F9E1EC48AA1","b":"7F9E1EC41000","o":"7AA1","s":"start_thread","s+":"D1"},{"a":"7F9E1E995C4D","b":"7F9E1E8AD000","o":"E8C4D","s":"clone","s+":"6D"}],"processInfo":{"mongodbVersion":"4.4.0-rc3-11-ge33cb0c","gitVersion":"e33cb0c81df6901645788f1df484c2d492c08b1f","compiledModules":["enterprise"],"uname":{"sysname":"Linux","release":"2.6.32-220.el6.x86_64","version":"#1 SMP Wed Nov 9 08:03:13 EST 2011","machine":"x86_64"},"somap":[{"b":"7F9E21ADC000","elfType":3,"buildId":"E0821A1C2652BA6E843F3CA225E16AFD84E6C26D"}]}}}

2. Manually construct the /version Evergreen URL https://evergreen.mongodb.com/version/mongodb_mongo_v4.4_e33cb0c81df6901645788f1df484c2d492c08b1f using the MongoDB version and githash.

3. Click on the compile task for the appropriate build variant (e.g. Enterprise RHEL 6.2 and https://evergreen.mongodb.com/task/mongodb_mongo_v4.4_enterprise_rhel_62_64_bit_compile_e33cb0c81df6901645788f1df484c2d492c08b1f_20_04_28_20_29_34). This step may be the most challenging because it requires interpreting the modules, distmod, platform, and architecture settings of the binary in the log output. SERVER-39859 would be valuable if we made it so this link automatically appeared in the "Files" section for the multiversion task.

4. Confirm the build-id of the executable.

$ curl https://s3.amazonaws.com/mciuploads/mongodb-mongo-v4.4/enterprise-rhel-62-64-bit/e33cb0c81df6901645788f1df484c2d492c08b1f/binaries/mongo-mongodb_mongo_v4.4_enterprise_rhel_62_64_bit_e33cb0c81df6901645788f1df484c2d492c08b1f_20_04_28_20_29_34.tgz -o binaries.tgz
$ tar zxvf binaries.tgz
$ file dist-test/bin/mongod
dist-test/bin/mongod: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 2.6.18, BuildID[sha1]=e0821a1c2652ba6e843f3ca225e16afd84e6c26d, not stripped

Comment by Eric Milkie [ 02/Mar/20 ]

This work could be useful as it would mean that we could also download the debug symbols for these binaries from Evergreen. Currently, if one of the older binaries crashes and leaves a core dump, we have no way of using it, since the binary and debug symbols are not saved.

Comment by Max Hirschhorn [ 27/Feb/19 ]

The "or possibly month-old binaries" was inaccurate and based on some confusion around the author date vs commit date. I think the worst we've seen is between a few hours and 1 day.

Generated at Thu Feb 08 04:53:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.