Tests are failing with OOM but evergreen is green

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: CI
    • 1
    • Not Needed
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Job example: https://spruce.mongodb.com/task/mongo_node_driver_next_rhel80_large_Node22_test_8.0_replica_set_patch_e5a85d0700592cac7c508932428fade337ead252_699718c74e2fe20007da5359_26_02_19_14_06_01/logs?execution=0
      It happens only on linux, the same task on windows failing with OOM correctly marked red.
      windows:
      https://spruce.mongodb.com/task/mongo_node_driver_next_windows_2022_latest_large_Node22_test_8.0_replica_set_patch_e5a85d0700592cac7c508932428fade337ead252_699718c74e2fe20007da5359_26_02_19_14_06_01/logs?execution=0

      The reason for this is bug (https://github.com/npm/cli/issues/6399) in NPM

      This has been fixed in newer version, and we can adopt changes by adopting devtools evegreen tools: https://github.com/mongodb-js/devtools-toolchain

      The reasons was classified incorrectly - we use latest NPM which catches exit code correctly. The real problem is nyc (tool we use for generating coverage), it swallows the exit code on linux, this is simple reproduction. Given: test that always OOM:

      describe.only('OOM exit code detection', function () {
  it('allocates until OOM', function () {
    const leak: object[] = [];
    while (true) {
      leak.push({ a: Array.from({ length: 1_000 }, (_, i) => ({ i })) });
    }
  });
});

      And different variants of invocation (direct mocha, mocha via npm, mocha via nyc, etc.):

      [2026/03/05 11:04:35.954]   os: 'linux'
[2026/03/05 11:01:57.366] node version: v25.8.0
[2026/03/05 11:01:57.435] npm version: 11.11.0
[2026/03/05 11:02:58.584] Layer 1: direct mocha
[2026/03/05 11:03:16.650] EXIT CODE: 134
[2026/03/05 11:03:16.650] Layer 2: npm run check:test (npm -> mocha)
[2026/03/05 11:03:39.390] EXIT CODE: 134
[2026/03/05 11:03:39.390] Layer 3: nyc mocha (nyc -> mocha, no npm)
[2026/03/05 11:04:01.140] EXIT CODE: 0
[2026/03/05 11:04:01.140] Layer 4: nyc npm run check:test (nyc -> npm -> mocha)
[2026/03/05 11:04:29.597] EXIT CODE: 0
[2026/03/05 11:04:29.597] Layer 5: npm run check:integration-coverage (npm -> nyc -> npm -> mocha)
[2026/03/05 11:04:56.306] EXIT CODE: 0

      Note how nyc swallowed exist code and the evergreen received 0 as exit code.
      Now windows:

      [2026/03/05 11:03:38.692]   os: 'win32'
[2026/03/05 11:01:58.461] node version: v24.14.0
[2026/03/05 11:01:58.928] npm version: 11.9.0
[2026/03/05 11:03:35.481] Layer 1: direct mocha
[2026/03/05 11:03:55.900] EXIT CODE: 134
[2026/03/05 11:03:55.900] Layer 2: npm run check:test (npm -> mocha)
[2026/03/05 11:04:21.300] EXIT CODE: 134
[2026/03/05 11:04:21.300] Layer 3: nyc mocha (nyc -> mocha, no npm)
[2026/03/05 11:04:45.662] EXIT CODE: 134
[2026/03/05 11:04:45.662] Layer 4: nyc npm run check:test (nyc -> npm -> mocha)
[2026/03/05 11:05:08.823] EXIT CODE: 134
[2026/03/05 11:05:08.823] Layer 5: npm run check:integration-coverage (npm -> nyc -> npm -> mocha)
[2026/03/05 11:05:32.977] EXIT CODE: 134

      Note how exit code (134) is constant on windows, and how nyc swallows it on linux. The problem most probably is not in nyc itself, but in module https://github.com/tapjs/foreground-child which wraps and handles processes for nyc.
      PR with the fix in upstream: https://github.com/tapjs/foreground-child/pull/71

            Assignee:
            Unassigned
            Reporter:
            Sergey Zelenov
            None
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: