[SERVER-27627] Use requests library to manage HTTP posts in resmoke.py buildlogger support Created: 10/Jan/17 Updated: 05/Apr/17 Resolved: 02/Mar/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | 3.5.4 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Andrew Morrow (Inactive) | Assignee: | Robert Guo (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | tig-resmoke | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Backport Requested: |
v3.4, v3.2
|
||||||||||||||||||||
| Sprint: | TIG 2017-03-06 | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Linked BF Score: | 0 | ||||||||||||||||||||
| Description |
|
It appears that urllib2 does no connection pooling, so every POST of log data to buildlogger requires a new HTTP connection. Moving to the requests (or some other more modern HTTP library) should allow pooling. |
| Comments |
| Comment by Githook User [ 01/Mar/17 ] | |||||||||||||||||||||||||||||||||
|
Author: {u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}Message: | |||||||||||||||||||||||||||||||||
| Comment by Githook User [ 28/Feb/17 ] | |||||||||||||||||||||||||||||||||
|
Author: {u'username': u'guoyr', u'name': u'Robert Guo', u'email': u'robert.guo@10gen.com'}Message: | |||||||||||||||||||||||||||||||||
| Comment by Sam Kleinman (Inactive) [ 24/Feb/17 ] | |||||||||||||||||||||||||||||||||
|
Insofar as I'm entitled to an opinion, I would say:
| |||||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 24/Feb/17 ] | |||||||||||||||||||||||||||||||||
|
Until Python 2.7.9, the urllib2 module didn't verify the certificate presented for HTTPS connections, so we could also choose to pass verify=False to requests.post() and suppress the warning from urllib3.
| |||||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 24/Feb/17 ] | |||||||||||||||||||||||||||||||||
|
I asked robert.guo to revert this change because logs aren't being sent to logkeeper as a result of the SSLError seen below. IIUC, the version of Python we use on many build variants doesn't support SNI, which causes the load balancer to present the certificate for .com rather than .org and for certificate validation to fail. CC ernie.hershey, sam.kleinman Edit: See also http://docs.python-requests.org/en/master/community/faq/#what-are-hostname-doesn-t-match-errors
Example task output
| |||||||||||||||||||||||||||||||||
| Comment by Githook User [ 24/Feb/17 ] | |||||||||||||||||||||||||||||||||
|
Author: {u'username': u'guoyr', u'name': u'Robert Guo', u'email': u'robert.guo@10gen.com'}Message: Revert " This reverts commit 14f16f384a2ace3b5ccb45dcbfbb66f3f57e945a. | |||||||||||||||||||||||||||||||||
| Comment by Githook User [ 24/Feb/17 ] | |||||||||||||||||||||||||||||||||
|
Author: {u'username': u'guoyr', u'name': u'Robert Guo', u'email': u'robert.guo@10gen.com'}Message: | |||||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 24/Jan/17 ] | |||||||||||||||||||||||||||||||||
|
We should also see if it is possible to use the requests package instead of shelling out to curl in the setup_multiversion_mongodb.py. (See |