[SERVER-80915] arm64 workstation and buildfarm container broken Created: 08/Sep/23  Updated: 27/Oct/23  Resolved: 27/Oct/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Daniel Moody Assignee: [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive)
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Server Development Platform
Operating System: ALL
Participants:

 Description   

I tried to setup a buildfarm container from a fresh spawn of the arm64 workstation, and there were several issues I encountered.

First the docker-compose deb package is not installed on the image, we should make a build ticket to have to incorporated on the images maybe?

Second there is a rogue "docker-compose" binary in the /usr/local/bin which will is not compatible with the docker compose yaml we have in buildfarm dir. It is first in the default path and so it even with docker-compose deb package installed it will be used instead of the one in /usr/bin. I am not sure if the /usr/local/bin one is used by someone else, but we should update our instructions maybe to be specific about /usr/bin/docker-compose?

Third, using the docker-compose in from the apt deb package, the current yaml is incorrect. You are not allow to use "ports" with "network_mode: host" so you must delete the port references. You get an error like this otherwise:

ubuntu@ip-10-128-18-126:~/mongo/buildfarm$ /usr/bin/docker-compose up --detach
Pulling redis (redis:5.0.9)...
5.0.9: Pulling from library/redis
4e6164a63b7b: Pull complete
66a6505e9bfd: Pull complete
fb4766a9fc53: Pull complete
1a4f5d531824: Pull complete
42a9344cf379: Pull complete
72f7ce5011d5: Pull complete
Digest: sha256:2a9865e55c37293b71df051922022898d8e4ec0f579c9b53a0caee1b170bc81c
Status: Downloaded newer image for redis:5.0.9
Creating buildfarm_redis_1 ...
 
ERROR: for buildfarm_redis_1  "host" network_mode is incompatible with port_bindings
 
ERROR: for redis  "host" network_mode is incompatible with port_bindings
Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 33, in <module>
    sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 203, in perform_command
    handler(command, command_options)
  File "/usr/lib/python3/dist-packages/compose/metrics/decorator.py", line 18, in wrapper
    result = fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1186, in up
    to_attach = up(False)
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1166, in up
    return self.project.up(
  File "/usr/lib/python3/dist-packages/compose/project.py", line 697, in up
    results, errors = parallel.parallel_execute(
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 108, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 206, in producer
    result = func(obj)
  File "/usr/lib/python3/dist-packages/compose/project.py", line 679, in do
    return service.execute_convergence_plan(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 559, in execute_convergence_plan
    return self._execute_convergence_create(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 473, in _execute_convergence_create
    containers, errors = parallel_execute(
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 108, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 206, in producer
    result = func(obj)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 478, in <lambda>
    lambda service_name: create_and_start(self, service_name.number),
  File "/usr/lib/python3/dist-packages/compose/service.py", line 457, in create_and_start
    container = service.create_container(number=n, quiet=True)
  File "/usr/lib/python3/dist-packages/compose/service.py", line 330, in create_container
    container_options = self._get_container_create_options(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 939, in _get_container_create_options
    container_options['host_config'] = self._get_container_host_config(
  File "/usr/lib/python3/dist-packages/compose/service.py", line 1014, in _get_container_host_config
    return self.client.create_host_config(
  File "/usr/lib/python3/dist-packages/docker/api/container.py", line 596, in create_host_config
    return HostConfig(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/docker/types/containers.py", line 336, in __init__
    raise host_config_incompatible_error(
docker.errors.InvalidArgument: "host" network_mode is incompatible with port_bindings

more info here: https://forums.docker.com/t/docker-errors-invalidargument-host-network-mode-is-incompatible-with-port-bindings/103492

you need to delete the port bindings fields to get it working:

(venv) ubuntu@ip-10-128-18-126:~/mongo$ git diff
diff --git a/buildfarm/docker-compose.yml b/buildfarm/docker-compose.yml
index 969033e6bca..834c7657471 100644
--- a/buildfarm/docker-compose.yml
+++ b/buildfarm/docker-compose.yml
@@ -4,9 +4,6 @@ services:
     build:
       context: .
       dockerfile: toolchain.dockerfile
-    ports:
-      # Note: This value must correspond to the value specified in `config.yml` for worker.publicName
-      - "8981:8981"
     depends_on:
       - redis
     network_mode: host
@@ -15,8 +12,6 @@ services:
     build:
       context: .
       dockerfile: toolchain.dockerfile
-    ports:
-      - "8980:8980"
     network_mode: host
     depends_on:
       - redis
@@ -24,6 +19,4 @@ services:
   redis:
     image: "redis:5.0.9"
     network_mode: host
-    ports:
-      # Note: This value must correspond to the value specified in `config.yml` for backplane.redisUri
-      - "6379:6379"
+


Generated at Thu Feb 08 06:44:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.