[CDRIVER-4150] AWS ECS tasks failing setup Created: 08/Sep/21  Updated: 28/Oct/23  Resolved: 25/Jul/22

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: None
Fix Version/s: 1.22.1

Type: Task Priority: Unknown
Reporter: Kevin Albertson Assignee: Kevin Albertson
Resolution: Fixed Votes: 0
Labels: failing-on-waterfall
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to JAVA-4401 run aws ECS auth test failing Closed
is related to DRIVERS-1462 Test Failure - MONGODB-AWS Auth Ubunt... Closed

 Description   

The test-aws-openssl-ecs-latest and test-aws-openssl-ecs-4.4 tasks are consistently failing to run the setup script aws_e2e_ecs.js.

Here is an example failure with relevant logs:

[2021/08/31 18:30:24.372] ===== Testing auth via ECS task metadata =====
[2021/08/31 18:30:24.372] Makefile
[2021/08/31 18:30:24.372] README.md
[2021/08/31 18:30:24.372] action.yml
[2021/08/31 18:30:24.372] evergreen_config_generator
[2021/08/31 18:30:24.440] MongoDB shell version v5.1.0-alpha-691-gc96a409
[2021/08/31 18:30:24.440] loading file: setup.js
[2021/08/31 18:30:24.440] loading file: aws_e2e_ecs.js
[2021/08/31 18:30:24.441] {"t":{"$date":"2021-08-31T18:30:24.441Z"},"s":"I",  "c":"-",        "id":22810,   "ctx":"js","msg":"shell: Started program","attr":{"pid":"8350","argv":["/bin/sh","-c","python3 -u  lib/container_tester.py -v remote_gc_services  --cluster arn:aws:ecs:us-east-1:557821124784:cluster/authtest-cluster"]}}
[2021/08/31 18:30:25.393] sh8350| AWS_SHARED_CREDENTIALS_FILE: None
[2021/08/31 18:30:25.393] sh8350| INFO:botocore.credentials:Found credentials in environment variables.
[2021/08/31 18:30:25.804] sh8350| INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): ecs.us-east-1.amazonaws.com
[2021/08/31 18:30:25.882] {"t":{"$date":"2021-08-31T18:30:25.882Z"},"s":"I",  "c":"-",        "id":22810,   "ctx":"js","msg":"shell: Started program","attr":{"pid":"8356","argv":["/bin/sh","-c","python3 -u  lib/container_tester.py -d -v run_e2e_test --cluster arn:aws:ecs:us-east-1:557821124784:cluster/authtest-cluster --task_definition arn:aws:ecs:us-east-1:557821124784:task-definition/app:2 --subnets subnet-6dd81326 --subnets subnet-50f3fe0a --security_group sg-09bacbe8dda5ccb7b --files /data/mci/33a57c2e3c4e7bb7bb053b284953a6bd/mongoc/mongodb/bin/mongod:/root/mongod /data/mci/33a57c2e3c4e7bb7bb053b284953a6bd/mongoc/mongodb/bin/mongo:/root/mongo  lib/ecs_hosted_test.js:/root/ecs_hosted_test.js /data/mci/33a57c2e3c4e7bb7bb053b284953a6bd/mongoc:/root --script lib/ecs_hosted_test.sh"]}}
[2021/08/31 18:30:26.084] sh8356| AWS_SHARED_CREDENTIALS_FILE: None
[2021/08/31 18:30:26.084] sh8356| DEBUG:botocore.credentials:Looking for credentials via: env
[2021/08/31 18:30:26.085] sh8356| INFO:botocore.credentials:Found credentials in environment variables.
[2021/08/31 18:30:26.085] sh8356| DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/endpoints.json
[2021/08/31 18:30:26.090] sh8356| DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/ecs/2014-11-13/service-2.json
[2021/08/31 18:30:26.092] sh8356| DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/_retry.json
[2021/08/31 18:30:26.094] sh8356| DEBUG:botocore.client:Registering retry handlers for service: ecs
[2021/08/31 18:30:26.094] sh8356| DEBUG:botocore.hooks:Event creating-client-class.ecs: calling handler <function add_generate_presigned_url at 0x7f48f992e488>
[2021/08/31 18:30:26.094] sh8356| DEBUG:botocore.args:The s3 config key is not a dictionary type, ignoring its value of: None
[2021/08/31 18:30:26.097] sh8356| DEBUG:botocore.endpoint:Setting ecs timeout as (60, 60)
[2021/08/31 18:30:26.097] sh8356| DEBUG:botocore.hooks:Event before-parameter-build.ecs.CreateService: calling handler <function generate_idempotent_uuid at 0x7f48f9865620>
[2021/08/31 18:30:26.097] sh8356| Traceback (most recent call last):
[2021/08/31 18:30:26.097] sh8356|   File "lib/container_tester.py", line 385, in <module>
[2021/08/31 18:30:26.098] sh8356|     main()
[2021/08/31 18:30:26.098] sh8356|   File "lib/container_tester.py", line 381, in main
[2021/08/31 18:30:26.098] sh8356|     args.func(args)
[2021/08/31 18:30:26.098] sh8356|   File "lib/container_tester.py", line 291, in _run_e2e_test_args
[2021/08/31 18:30:26.098] sh8356|     _run_e2e_test(args.script, args.files, args.cluster, args.task_definition, args.subnets, args.security_group)
[2021/08/31 18:30:26.098] sh8356|   File "lib/container_tester.py", line 303, in _run_e2e_test
[2021/08/31 18:30:26.098] sh8356|     remote_create_container(cluster, task_definition, service_name, subnets, security_group)
[2021/08/31 18:30:26.098] sh8356|   File "lib/container_tester.py", line 170, in remote_create_container
[2021/08/31 18:30:26.098] sh8356|     'assignPublicIp': "ENABLED"
[2021/08/31 18:30:26.098] sh8356|   File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 253, in _api_call
[2021/08/31 18:30:26.185] sh8356|     return self._make_api_call(operation_name, kwargs)
[2021/08/31 18:30:26.185] sh8356|   File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 531, in _make_api_call
[2021/08/31 18:30:26.186] sh8356|     api_params, operation_model, context=request_context)
[2021/08/31 18:30:26.186] sh8356|   File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 586, in _convert_to_request_dict
[2021/08/31 18:30:26.186] sh8356|     api_params, operation_model)
[2021/08/31 18:30:26.186] sh8356|   File "/usr/local/lib/python3.6/dist-packages/botocore/validate.py", line 291, in serialize_to_request
[2021/08/31 18:30:26.186] sh8356|     raise ParamValidationError(report=report.generate_report())
[2021/08/31 18:30:26.186] sh8356| botocore.exceptions.ParamValidationError: Parameter validation failed:
[2021/08/31 18:30:26.186] sh8356| Unknown parameter in input: "launchType", must be one of: cluster, serviceName, taskDefinition, loadBalancers, desiredCount, clientToken, role, deploymentConfiguration, placementConstraints, placementStrategy
[2021/08/31 18:30:26.186] sh8356| Unknown parameter in input: "networkConfiguration", must be one of: cluster, serviceName, taskDefinition, loadBalancers, desiredCount, clientToken, role, deploymentConfiguration, placementConstraints, placementStrategy
[2021/08/31 18:30:26.214] uncaught exception: Error: [1] != [0] are not equal : Container Test failed :
[2021/08/31 18:30:26.214] doassert@src/mongo/shell/assert.js:20:14
[2021/08/31 18:30:26.214] assert.eq@src/mongo/shell/assert.js:179:9
[2021/08/31 18:30:26.214] @aws_e2e_ecs.js:43:4
[2021/08/31 18:30:26.214] @aws_e2e_ecs.js:6:2
[2021/08/31 18:30:26.220] failed to load: aws_e2e_ecs.js
[2021/08/31 18:30:26.220] exiting with code -3
[2021/08/31 18:30:26.220] Command failed: command encountered problem: error waiting on process '025152fe-bd6e-446c-b9e9-767787eb67bc': exit status 253
[2021/08/31 18:30:26.220] Task completed - FAILURE.



 Comments   
Comment by Githook User [ 25/Jul/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: CDRIVER-4150 fix the AWS ECS test tasks (#1073)

  • Use a Python virtual environment with upgraded boto3.
  • Use separate client in shell auth.
Comment by Githook User [ 25/Jul/22 ]

Author:

{'name': 'Kevin Albertson', 'email': 'kevin.albertson@mongodb.com', 'username': 'kevinAlbs'}

Message: CDRIVER-4150 fix the AWS ECS test tasks (#1073)

  • Use a Python virtual environment with upgraded boto3.
  • Use separate client in shell auth.
Generated at Wed Feb 07 21:20:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.