-
Type: Bug
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Testing
-
None
Tests on Travis macOS appear to be very susceptible to timing issues recently. Since we continuously test with macOS 10.14 on Evergreen, and GitHub PRs can be easily run as Evergreen patch builds, there does not appear to be a compelling reason to include macOS in Travis.
On master, the first appearance I see is this commit: https://travis-ci.org/mongodb/mongo-c-driver/jobs/625807390?utm_medium=notification&utm_source=github_status
1: Assert Failure: 324 not within 50% of 200 1: /Users/travis/build/mongodb/mongo-c-driver/src/libmongoc/tests/test-mongoc-topology.c:862 _test_select_succeed() 1: { "status": "fail", "test_file": "/Topology/try_once/succeed", "seed": "2776755316", "start": 710.855521, "end": 730.830371, "elapsed": 19.974850 },
Another, different test:
https://travis-ci.org/mongodb/mongo-c-driver/jobs/629764378?utm_medium=notification&utm_source=github_status
1: Assert Failure: 309 not within 50% of 200 1: /Users/travis/build/mongodb/mongo-c-driver/src/libmongoc/tests/test-mongoc-topology.c:862 _test_select_succeed() 1: { "status": "fail", "test_file": "/Topology/connect_timeout/succeed", "seed": "3992743056", "start": 701.108210, "end": 715.855158, "elapsed": 14.746948 },
Here is a pretty isolated case: https://travis-ci.org/mongodb/mongo-c-driver/jobs/648935093#L3121
1: expected to wait 100ms, waited 234430 1: { "status": "pass", "test_file": "/Thread/cond_wait", "seed": "544052547", "start": 876.469960, "end": 876.875507, "elapsed": 0.405547 },
That comes from this test: https://github.com/mongodb/mongo-c-driver/blob/1.16.0/src/libmongoc/tests/test-mongoc-thread.c/#L22-L24
The 234430 is microseconds, so the test really took 234 milliseconds. All it does is call mongoc_cond_timedwait with 100ms and assert it expires in the range 50-150ms. mongoc_cond_timedwait is a light wrapper around pthread_cond_timedwait. It seems highly unlikely any real bug exists there.
Comparing the "Build System Information" of the logs of Travis builds after they started failing frequently, it appears the default macOS version was bumped from 10.13.3 to 10.13.6.
Before: https://travis-ci.org/mongodb/mongo-c-driver/jobs/624818762?utm_medium=notification&utm_source=github_status
After: https://travis-ci.org/mongodb/mongo-c-driver/jobs/625807390?utm_medium=notification&utm_source=github_status
The downside of removing macOS is that it makes it more difficult for contributors to test with macOS, as it requires a MongoDB employee to trigger Evergreen patch build on a PR. We can revisit this if it a need arises. But until then, let's aim to have Travis passing so we don't start to ignore when it fails.