Uploaded image for project: 'C Driver'
  1. C Driver
  2. CDRIVER-3520

Remove macOS from Travis testing

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 1.17.0-beta, 1.17.0
    • Affects Version/s: None
    • Component/s: Testing
    • Labels:
      None

      Tests on Travis macOS appear to be very susceptible to timing issues recently. Since we continuously test with macOS 10.14 on Evergreen, and GitHub PRs can be easily run as Evergreen patch builds, there does not appear to be a compelling reason to include macOS in Travis.

      On master, the first appearance I see is this commit: https://travis-ci.org/mongodb/mongo-c-driver/jobs/625807390?utm_medium=notification&utm_source=github_status

      1: Assert Failure: 324 not within 50% of 200
      1: /Users/travis/build/mongodb/mongo-c-driver/src/libmongoc/tests/test-mongoc-topology.c:862  _test_select_succeed()
      1:     { "status": "fail", "test_file": "/Topology/try_once/succeed", "seed": "2776755316", "start": 710.855521, "end": 730.830371, "elapsed": 19.974850  },
      

      Another, different test:
      https://travis-ci.org/mongodb/mongo-c-driver/jobs/629764378?utm_medium=notification&utm_source=github_status

      1: Assert Failure: 309 not within 50% of 200
      1: /Users/travis/build/mongodb/mongo-c-driver/src/libmongoc/tests/test-mongoc-topology.c:862  _test_select_succeed()
      1:     { "status": "fail", "test_file": "/Topology/connect_timeout/succeed", "seed": "3992743056", "start": 701.108210, "end": 715.855158, "elapsed": 14.746948  },
      

      Here is a pretty isolated case: https://travis-ci.org/mongodb/mongo-c-driver/jobs/648935093#L3121

      1: expected to wait 100ms, waited 234430
      1:     { "status": "pass", "test_file": "/Thread/cond_wait", "seed": "544052547", "start": 876.469960, "end": 876.875507, "elapsed": 0.405547  },
      

      That comes from this test: https://github.com/mongodb/mongo-c-driver/blob/1.16.0/src/libmongoc/tests/test-mongoc-thread.c/#L22-L24
      The 234430 is microseconds, so the test really took 234 milliseconds. All it does is call mongoc_cond_timedwait with 100ms and assert it expires in the range 50-150ms. mongoc_cond_timedwait is a light wrapper around pthread_cond_timedwait. It seems highly unlikely any real bug exists there.

      Comparing the "Build System Information" of the logs of Travis builds after they started failing frequently, it appears the default macOS version was bumped from 10.13.3 to 10.13.6.
      Before: https://travis-ci.org/mongodb/mongo-c-driver/jobs/624818762?utm_medium=notification&utm_source=github_status
      After: https://travis-ci.org/mongodb/mongo-c-driver/jobs/625807390?utm_medium=notification&utm_source=github_status

      The downside of removing macOS is that it makes it more difficult for contributors to test with macOS, as it requires a MongoDB employee to trigger Evergreen patch build on a PR. We can revisit this if it a need arises. But until then, let's aim to have Travis passing so we don't start to ignore when it fails.

            Assignee:
            kevin.albertson@mongodb.com Kevin Albertson
            Reporter:
            kevin.albertson@mongodb.com Kevin Albertson
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: