Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-3102

Add backtracking support for MongoDB downloads

    • Not Needed

      We recently had a situation where there were partial downloads available for 8.0.5-rc0.  In this case downloads of "latest" resulted in a 403 error:

      https://spruce.mongodb.com/task/mongo_c_driver_latest_release_cse_matrix_darwinssl_cse_sasl_cyrus_darwinssl_macos_1100_clang_test_4.4_server_auth_ba3461e107e30fa88632cd14b8da88bd05ec1e18_25_02_04_14_23_45/logs?execution=0

       

      [2025/02/04 08:33:48.623] INFO     Download crypt_shared 4.4-enterprise for macos-x86_64[2025/02/04 08:33:48.623] /Users/ec2-user/data/mci/a17dc4d86d400e9b2061cdd809ba40b7/drivers-evergreen-tools/.local/uv-tool/drivers-orchestration/lib/python3.13/site-packages/mongodl.py:1161: UserWarning: No matching version of crypt_shared found, using 'latest'[2025/02/04 08:33:48.623]   result = _dl_component([2025/02/04 08:33:48.655] Traceback (most recent call last):[2025/02/04 08:33:48.656]   File "/Users/ec2-user/data/mci/a17dc4d86d400e9b2061cdd809ba40b7/drivers-evergreen-tools/.local/uv-tool/drivers-orchestration/lib/python3.13/site-packages/mongodl.py", line 597, in download_file[2025/02/04 08:33:48.656]     resp = urllib.request.urlopen(req, context=SSL_CONTEXT)[2025/02/04 08:33:48.656]   File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/urllib/request.py", line 189, in urlopen[2025/02/04 08:33:48.656]     return opener.open(url, data, timeout)[2025/02/04 08:33:48.656]            ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^[2025/02/04 08:33:48.656]   File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/urllib/request.py", line 495, in open[2025/02/04 08:33:48.656]     response = meth(req, response)[2025/02/04 08:33:48.656]   File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/urllib/request.py", line 604, in http_response[2025/02/04 08:33:48.656]     response = self.parent.error([2025/02/04 08:33:48.656]         'http', request, response, code, msg, hdrs)[2025/02/04 08:33:48.656]   File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/urllib/request.py", line 533, in error[2025/02/04 08:33:48.656]     return self._call_chain(*args)[2025/02/04 08:33:48.656]            ~~~~~~~~~~~~~~~~^^^^^^^[2025/02/04 08:33:48.656]   File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/urllib/request.py", line 466, in _call_chain[2025/02/04 08:33:48.656]     result = func(*args)[2025/02/04 08:33:48.656]   File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/urllib/request.py", line 613, in http_error_default[2025/02/04 08:33:48.656]     raise HTTPError(req.full_url, code, msg, hdrs, fp)[2025/02/04 08:33:48.656] urllib.error.HTTPError: HTTP Error 403: Forbidden[2025/02/04 08:33:48.656] The above exception was the direct cause of the following exception:[2025/02/04 08:33:48.656] Traceback (most recent call last):[2025/02/04 08:33:48.656]   File "/Users/ec2-user/data/mci/a17dc4d86d400e9b2061cdd809ba40b7/drivers-evergreen-tools/.evergreen/orchestration/drivers-orchestration", line 12, in <module>[2025/02/04 08:33:48.656]     sys.exit(main())[2025/02/04 08:33:48.656]              ~~~~^^[2025/02/04 08:33:48.656]   File "/Users/ec2-user/data/mci/a17dc4d86d400e9b2061cdd809ba40b7/drivers-evergreen-tools/.evergreen/orchestration/drivers_orchestration.py", line 434, in main[2025/02/04 08:33:48.656]     run(opts)[2025/02/04 08:33:48.656]     ~~~^^^^^^[2025/02/04 08:33:48.656]   File "/Users/ec2-user/data/mci/a17dc4d86d400e9b2061cdd809ba40b7/drivers-evergreen-tools/.evergreen/orchestration/drivers_orchestration.py", line 223, in run[2025/02/04 08:33:48.656]     mongodl(shlex.split(args))[2025/02/04 08:33:48.657]     ~~~~~~~^^^^^^^^^^^^^^^^^^^[2025/02/04 08:33:48.657]   File "/Users/ec2-user/data/mci/a17dc4d86d400e9b2061cdd809ba40b7/drivers-evergreen-tools/.local/uv-tool/drivers-orchestration/lib/python3.13/site-packages/mongodl.py", line 1161, in main[2025/02/04 08:33:48.657]     result = _dl_component([2025/02/04 08:33:48.657]         cache,[2025/02/04 08:33:48.657]     ...<10 lines>...[2025/02/04 08:33:48.657]         latest_build_branch=args.latest_build_branch,[2025/02/04 08:33:48.657]     )[2025/02/04 08:33:48.657]   File "/Users/ec2-user/data/mci/a17dc4d86d400e9b2061cdd809ba40b7/drivers-evergreen-tools/.local/uv-tool/drivers-orchestration/lib/python3.13/site-packages/mongodl.py", line 853, in _dl_component[2025/02/04 08:33:48.657]     cached = cache.download_file(dl_url).path[2025/02/04 08:33:48.657]              ~~~~~~~~~~~~~~~~~~~^^^^^^^^[2025/02/04 08:33:48.657]   File "/Users/ec2-user/data/mci/a17dc4d86d400e9b2061cdd809ba40b7/drivers-evergreen-tools/.local/uv-tool/drivers-orchestration/lib/python3.13/site-packages/mongodl.py", line 600, in download_file[2025/02/04 08:33:48.657]     raise RuntimeError(f"Failed to download [{url}]") from e[2025/02/04 08:33:48.657] RuntimeError: Failed to download [https://downloads.mongodb.com/osx/mongo_crypt_shared_v1-macos-x86_64-enterprise-8.0.5-rc0.tgz] 

       

      For versions such as "rapid", "latest", "8.0", etc., we should improve robustness by backtracking to find an available download.

            Assignee:
            steve.silvester@mongodb.com Steve Silvester
            Reporter:
            steve.silvester@mongodb.com Steve Silvester
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: