Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-70839

Spawning dynamically linked mongod processes takes ~30 secs on EVG

XMLWordPrintableJSON

    • Type: Icon: Question Question
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • Server Development Platform

      Motivation for Request

      Reduce considerably core server EVG patch time.

      Context

      As part of my Skunkwork project I tried to understand why starting up a sharded cluster using ShardingTest takes so long. Surprisingly I discovered that just spawning the mongod processes can take up to ~30 secs on Evegreen.

      With "spawning" I mean the time that pass from when we [call execve| we call execve] to the time we execute the first instruction in mongod_main.
      For instance in this patch it took ~34 secs to spawn 1 mongod process:

      [js_test:shardingtest_control_12_nodes] d22050| XOXO execve at time: Mon Jul 18 14:51:16 2022
[js_test:shardingtest_control_12_nodes] d22050| XOXO mongod_main start: Mon Jul 18 14:51:25 2022

      As of today we run ~883 tests in every of the 10 sharding suites, each test take ~30 secs just to spawn the process required to setup the cluster. So only accounting for sharding tests, we are spending at least 73 AWS machine hours per patch per variant just to spawn those processes.

      It seems we have a huge space for improvement here.

            Assignee:
            backlog-server-devplatform [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive)
            Reporter:
            tommaso.tocci@mongodb.com Tommaso Tocci
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: