[SERVER-28401] Use Python's requests package in setup_multiversion_mongodb.py instead of curl Created: 20/Mar/17  Updated: 06/Dec/17  Resolved: 18/Oct/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.6.0-rc1

Type: Task Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Jonathan Abrahams
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-22150 multiversion download script should u... Closed
is related to SERVER-27251 setup_multiversion_mongodb.py should ... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2017-10-23
Participants:

 Description   

Due to versions of Python < 2.7.9 not supporting certificate verification, the setup_multiversion_mongodb.py script was changed to download the feed and binaries using curl. This makes it more difficult to implement additional retry logic on top of curl. We should implement additional retry logic by using Python's requests package and logic similar to that in buildscripts/resmokelib/logging/handlers.py to tolerate older versions. SERVER-27251 can likely be closed a duplicate of this ticket if the setup_multiversion_mongodb.py script retries on any exception when running requests.get() and requests.Response.iter_content().

https://jira.mongodb.org/browse/SERVER-27627?focusedCommentId=1508925&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1508925



 Comments   
Comment by Githook User [ 18/Oct/17 ]

Author:

{'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams', 'username': 'hptabster'}

Message: SERVER-28403 setup_multiversion_mongodb.py looks for latest when downloading Major.minor
SERVER-27251 setup_multiversion_mongodb.py should retry in the case of failures
SERVER-28401 setup_multiversion_mongodb.py uses requests package for downloads
Branch: master
https://github.com/mongodb/mongo/commit/18d5b0cea7558f88bbd5dcbec2a762b51cb13c98

Comment by Jonathan Abrahams [ 18/Oct/17 ]

Work for this ticket was subsumed in SERVER-28403

Comment by Jonathan Abrahams [ 09/Oct/17 ]

Workaround is

pip install pyOpenSSL

Comment by Jonathan Abrahams [ 09/Oct/17 ]

We are using an old version of OpenSSL, 0.9.81 in Windows python:

>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 0.9.8l 5 Nov 2009'

We need to upgrade python/OpenSSL, see https://stackoverflow.com/questions/38501531/forcing-requests-library-to-use-tlsv1-1-or-tlsv1-2-in-python and https://stackoverflow.com/questions/18752409/updating-openssl-in-python-2-7

Comment by Jonathan Abrahams [ 09/Oct/17 ]

When using python 2.7.3 on WIndows, with requests 2.18.4 package, the URL that we are downloading https://downloads.mongodb.org/full.json fails on the SSL handshake:

requests.exceptions.SSLError: HTTPSConnectionPool(host='downloads.mongodb.org', port=443): Max retries exceeded with url: /full.json (Caused by SSLError(SSLError(1, '_ssl.c:504: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure'),))

curl -v -I  'https://downloads.mongodb.org/full.json'
* STATE: INIT => CONNECT handle 0x600057830; line 1410 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* STATE: CONNECT => WAITRESOLVE handle 0x600057830; line 1446 (connection #0)
*   Trying 54.230.19.119...
* TCP_NODELAY set
* STATE: WAITRESOLVE => WAITCONNECT handle 0x600057830; line 1527 (connection #0)
* Connected to downloads.mongodb.org (54.230.19.119) port 443 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x600057830; line 1579 (connection #0)
* Marked for [keep alive]: HTTP default
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x600057830; line 1593 (connection #0)
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256

Without a solution in python 2.7.3 it might more sense to use python3 for this.

Comment by Jonathan Abrahams [ 02/Oct/17 ]

This is being done with work for SERVER-28403.

Generated at Thu Feb 08 04:18:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.