[CDRIVER-3520] Remove macOS from Travis testing Created: 03/Feb/20  Updated: 28/Oct/23  Resolved: 13/Feb/20

Status: Closed
Project: C Driver
Component/s: tests
Affects Version/s: None
Fix Version/s: 1.17.0-beta, 1.17.0

Type: Bug Priority: Major - P3
Reporter: Kevin Albertson Assignee: Kevin Albertson
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Tests on Travis macOS appear to be very susceptible to timing issues recently. Since we continuously test with macOS 10.14 on Evergreen, and GitHub PRs can be easily run as Evergreen patch builds, there does not appear to be a compelling reason to include macOS in Travis.

On master, the first appearance I see is this commit: https://travis-ci.org/mongodb/mongo-c-driver/jobs/625807390?utm_medium=notification&utm_source=github_status

1: Assert Failure: 324 not within 50% of 200
1: /Users/travis/build/mongodb/mongo-c-driver/src/libmongoc/tests/test-mongoc-topology.c:862  _test_select_succeed()
1:     { "status": "fail", "test_file": "/Topology/try_once/succeed", "seed": "2776755316", "start": 710.855521, "end": 730.830371, "elapsed": 19.974850  },

Another, different test:
https://travis-ci.org/mongodb/mongo-c-driver/jobs/629764378?utm_medium=notification&utm_source=github_status

1: Assert Failure: 309 not within 50% of 200
1: /Users/travis/build/mongodb/mongo-c-driver/src/libmongoc/tests/test-mongoc-topology.c:862  _test_select_succeed()
1:     { "status": "fail", "test_file": "/Topology/connect_timeout/succeed", "seed": "3992743056", "start": 701.108210, "end": 715.855158, "elapsed": 14.746948  },

Here is a pretty isolated case: https://travis-ci.org/mongodb/mongo-c-driver/jobs/648935093#L3121

1: expected to wait 100ms, waited 234430
1:     { "status": "pass", "test_file": "/Thread/cond_wait", "seed": "544052547", "start": 876.469960, "end": 876.875507, "elapsed": 0.405547  },

That comes from this test: https://github.com/mongodb/mongo-c-driver/blob/1.16.0/src/libmongoc/tests/test-mongoc-thread.c/#L22-L24
The 234430 is microseconds, so the test really took 234 milliseconds. All it does is call mongoc_cond_timedwait with 100ms and assert it expires in the range 50-150ms. mongoc_cond_timedwait is a light wrapper around pthread_cond_timedwait. It seems highly unlikely any real bug exists there.

Comparing the "Build System Information" of the logs of Travis builds after they started failing frequently, it appears the default macOS version was bumped from 10.13.3 to 10.13.6.
Before: https://travis-ci.org/mongodb/mongo-c-driver/jobs/624818762?utm_medium=notification&utm_source=github_status
After: https://travis-ci.org/mongodb/mongo-c-driver/jobs/625807390?utm_medium=notification&utm_source=github_status

The downside of removing macOS is that it makes it more difficult for contributors to test with macOS, as it requires a MongoDB employee to trigger Evergreen patch build on a PR. We can revisit this if it a need arises. But until then, let's aim to have Travis passing so we don't start to ignore when it fails.



 Comments   
Comment by Githook User [ 12/Feb/20 ]

Author:

{'name': 'Kevin Albertson', 'username': 'kevinAlbs', 'email': 'kevin.albertson@mongodb.com'}

Message: CDRIVER-3520 remove macOS from Travis
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/db95c08ee3fcee664e4414515cbff8a06c872f2c

Generated at Wed Feb 07 21:18:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.