[DRIVERS-2648] Add driver tests for Serverless Proxy incremental rollout Created: 12/Jun/23 Updated: 25/Jan/24 |
|
| Status: | Implementing |
| Project: | Drivers |
| Component/s: | Serverless |
| Fix Version/s: | None |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Siyuan Zhou | Assignee: | Steve Silvester |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Driver Changes: | Needed - No Spec Changes | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Quarter: | FY24Q4 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Downstream Changes Summary: | Cloud is implementing a new serverless proxy that will be incrementally rolled out to new and existing clusters over the coming months. The team is leveraging pymongo's test suite to verify compliance with driver specs. In order to help ensure compatibility with all of our drivers - especially where they differ subtly and unintentionally from our official specs - we have agreed to add tests against the new serverless proxy to each driver's CI suite. There are known failures that are being tracked in DRIVERS-2812. The Serverless Proxy tests should not be run on PR commits until those are addressed.
Each driver must continue running their serverless test suite against the existing serverless project (which will remain configured for the current serverless proxy). Each driver must run that same serverless test suite against the new serverless project which has been configured to use the new serverless proxy. Please let james.kovacs@mongodb.com (DBX) and siyuan.zhou@mongodb.com (Serverless) know of any reproducible failures against the new serverless proxy. Note for implementers: there is a new SERVERLESS_REGION variable that must be set to US_EAST_1. It is recommended that drivers follow the pattern used in https://github.com/mongodb/mongo-python-driver/commit/41a131ea1c15ffa969a14ce1334ce19837dc226b, as described in the Serverless README. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Start date: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Driver Compliance: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
| Comments |
| Comment by Githook User [ 22/Jan/24 ] |
|
Author: {'name': 'Steven Silvester', 'email': 'steven.silvester@ieee.org', 'username': 'blink1073'}Message: DRIVERS-2648 Add AWS Secrets handling for serverless (#371)
|
| Comment by Githook User [ 19/Jan/24 ] |
|
Author: {'name': 'Steven Silvester', 'email': 'steven.silvester@ieee.org', 'username': 'blink1073'}Message: DRIVERS-2648 Add AWS Secrets handling for serverless (#371)
|
| Comment by Celina Tala [ 19/Jan/24 ] |
|
Final Update: All but 2 driver tests are successfully running under gRPC. test_errors_during_the_initial_connection_hello_are_ignored: This is to be expected as we currently handle the hello command in the serverless proxy instead of the atlas proxy, while failpoints are counted in the atlas proxy. It will be resolved once we move failpoint configuration to the serverless proxy.
test_pinned_connection_is_released_after_a_transient_network_commit_error: This error is also expected until PYTHON-4074 is resolved. |
| Comment by Celina Tala [ 18/Jan/24 ] |
|
Filed |
| Comment by Steve Silvester [ 12/Jan/24 ] |
|
Running in US_EAST_1 we're now down to three failures. I'm going to consider this unblocked from the drivers end as the serverless team works on the remaining errors. |
| Comment by Celina Tala [ 11/Jan/24 ] |
|
steve.silvester@mongodb.com We realized that the drivers test creates serverless instances in region US_EAST_2 (here) while our gRPC MTM is created in region US_EAST_1. Would it be easy to change this script? We can also make a new MTM in the correct region on our side. We also had to turn on Single Target Serverless Deployment and Atlas Enable Test Commands feature flag for our gRPC MTM Group |
| Comment by Celina Tala [ 10/Jan/24 ] |
|
steve.silvester@mongodb.com Hi steve! I looked into the failing tests in the patch you linked, and all of them succeeded in our nightly test runs (the specific tests are bookmarked). The test_errors_during_the_initial_connection_hello_are_ignored test is expected to fail as our hello command is dealt with in the serverless proxy while failpoints are still dealt with in our Atlas Proxy. So I wonder if the failures come from somewhere else? Our test currently runs Python Driver version 4.6.0 using python 3.11. I see you mentioned that multiple drivers are also experiencing test failures, are the tests all the same across all drivers? |
| Comment by Steve Silvester [ 10/Jan/24 ] |
|
Hi tristan.wedderburn@mongodb.com, we don't have anything merged yet, since it isn't fully working in any of the drivers. Here is the open PR for Python: https://github.com/mongodb/mongo-python-driver/pull/1428. And the latest manual build, with the same set of failures as when I last tried in December: |
| Comment by Steve Silvester [ 10/Jan/24 ] |
|
Hi tristan.wedderburn@mongodb.com, we don't have anything merged yet, since it isn't fully working in any of the drivers. Here is the open PR for Python: https://github.com/mongodb/mongo-python-driver/pull/1428. And the latest manual build, with the same set of failures as when I last tried in December: |
| Comment by Tristan Wedderburn [ 10/Jan/24 ] |
|
Hi steve.silvester@mongodb.com, Siyuan has released a newer version to cloud-dev as of today. Is there a nightly evergreen patch that we can watch? |
| Comment by Steve Silvester [ 08/Jan/24 ] |
|
Hi tristan.wedderburn@mongodb.com, I'm back in office. I started watching CLOUDP-219103, is the plan to release it this week? |
| Comment by Steve Silvester [ 21/Dec/23 ] |
|
Thanks tristan.wedderburn@mongodb.com! Today is my last day in office for the next two weeks, we can sync up when I get back. |
| Comment by Tristan Wedderburn [ 21/Dec/23 ] |
|
Hi Team, after investigating some of the recent driver test failures, it appears that we need to release a newer version of the serverless proxy to cloud-dev. For example, we need to release functionality such as supporting C# driver auth over OP_QUERY, which was added in CLOUDP-201831. Since this is a short week, we will proceed with the upgrade early next week. CLOUDP-219103 will track that upgrade. Once that is released, we should revisit the remaining failing tests across the drivers |
| Comment by Siyuan Zhou [ 20/Dec/23 ] |
|
steve.silvester@mongodb.com, thank you! tristan.wedderburn@mongodb.com can help you debugging the failures this week. |
| Comment by Steve Silvester [ 18/Dec/23 ] |
|
siyuan.zhou@mongodb.com, celina.tala@mongodb.com, given that we are failing the same tests against multiple drivers with the new proxy version, what steps would you like to take next? |
| Comment by Celina Tala [ 16/Nov/23 ] |
|
Also fyi we still have not turned on gRPC for these tests yet. We will let you know once we do |