-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Server Programmability
-
Programmability 2025-10-27
-
None
-
None
-
None
-
None
-
None
-
None
-
None
From slack:
Hi, I figured out that when using the dev_stacktrace option, many of the C++ unit tests on current master take significantly longer to execute than when not using dev_stacktrace.
The problem seem to be the "death tests", which intentionally make the process crash with invariant(...) failures or similar measures. These death tests seem to be significantly slowed down when using dev_stacktrace. In my local environment (AWS arm64 buildhost running Ubuntu 22.04), the death tests take 9x as long with dev_stacktrace than without.For example, here are the test outputs for the same set of tests, once run with dev_stacktrace and once without. This shows the timing difference:
$ bazel test --//bazel/config:dev_stacktrace=True +change_stream_test {"t":{"$date":"2025-10-28T13:42:56.573+00:00"},"s":"I", "c":"-","id":8991200, "ctx":"main","msg":"Shuffling initializers","attr":{"seed":3750308923}} ... {"t":{"$date":"2025-10-28T13:45:21.156+00:00"},"s":"I", "c":"TEST","id":23065, "ctx":"main","msg":"Totals","attr":{"totals":{"name":"TOTALS", "tests":220,"fails":0,"time":{"durationMillis":144486},"failures":[]}}} {"t":{"$date":"2025-10-28T13:45:21.156+00:00"},"s":"I", "c":"TEST","id":23069, "ctx":"main","msg":"SUCCESS - All tests in all suites passed"} //src/mongo/db/pipeline:db_pipeline_test PASSED in 144.8s $ bazel test --//bazel/config:dev_stacktrace=False +change_stream_test {"t":{"$date":"2025-10-28T13:48:53.957+00:00"},"s":"I", "c":"-","id":8991200, "ctx":"main","msg":"Shuffling initializers","attr":{"seed":1399599723}} ... {"t":{"$date":"2025-10-28T13:49:17.758+00:00"},"s":"I", "c":"TEST","id":23065, "ctx":"main","msg":"Totals","attr":{"totals":{"name":"TOTALS", "tests":220,"fails":0,"time":{"durationMillis":23706},"failures":[]}}} {"t":{"$date":"2025-10-28T13:49:17.759+00:00"},"s":"I", "c":"TEST","id":23069, "ctx":"main","msg":"SUCCESS - All tests in all suites passed"} //src/mongo/db/pipeline:db_pipeline_test PASSED in 24.0s
Non-death tests do not seem to be affected by the slowdown. This makes sense to me, because dev_stacktrace is a backtrace symbolizer that is likely only invoked upon invariant(...) failures, tasserts and the like.I am not sure if this has been reported before, so I wanted to share this finding now. The slowdown also seems to be the reason why I cannot successfully run larger unit tests (e.g. db_pipeline_test), because they now always run into the 300s timeout before completing.
- depends on
-
-
- Open
-