[SERVER-29507] Task not marked as failure if resmoke.py fails to start fixtures Created: 07/Jun/17  Updated: 30/Oct/23  Resolved: 09/Jun/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.5.9

Type: Bug Priority: Critical - P2
Reporter: Charlie Swanson Assignee: Eddie Louie
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-28784 Update "run tests" function to run un... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: TIG 2017-06-19
Participants:

 Description   

Happened in one of my patch builds. All mongod's in this patch fail to start with an error message like this:

[MongoDFixture:job2] Starting mongod on port 20500...
./mongod --setParameter enableTestCommands=1 --dbpath /data/db/job2/resmoke --nopreallocj --storageEngine wiredTiger --wiredTigerCacheSizeGB 1 --bind_ip_all --port 20500
[MongoDFixture:job2] mongod started on port 20500 with pid 6508.
[MongoDFixture:job2] src/third_party/timelib-2017.05beta3/parse_tz.c:152:16: runtime error: left shift of 128 by 24 places cannot be represented in type 'int'
[MongoDFixture:job2]     #0 0x556486bc1ca2 in read_transistions /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/third_party/timelib-2017.05beta3/parse_tz.c:152:16
[MongoDFixture:job2]     #1 0x556486bc1ca2 in timelib_parse_tzfile /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/third_party/timelib-2017.05beta3/parse_tz.c:466
[MongoDFixture:job2]     #2 0x556486bbe529 in mongo::DateTimeSupport::loadTimeZoneInfo(std::unique_ptr<_timelib_tzdb, mongo::DateTimeSupport::TimeZoneDBDeleter>) /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/db/query/datetime/date_time_support.cpp:101:23
[MongoDFixture:job2]     #3 0x556486bbe3c9 in mongo::DateTimeSupport::DateTimeSupport() /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/db/query/datetime/date_time_support.cpp:71:5
[MongoDFixture:job2]     #4 0x556486873fc7 in std::_MakeUniq<mongo::DateTimeSupport>::__single_object std::make_unique<mongo::DateTimeSupport>() /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/bits/unique_ptr.h:765:34
[MongoDFixture:job2]     #5 0x556486873fc7 in mongo::_mongoInitializerFunction_LoadTimeZoneDB(mongo::InitializerContext*) /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/db/query/datetime/init_timezone_data.cpp:59
[MongoDFixture:job2]     #6 0x556485762a3f in std::_Function_handler<mongo::Status (mongo::InitializerContext*), mongo::Status (*)(mongo::InitializerContext*)>::_M_invoke(std::_Any_data const&, mongo::InitializerContext*&&) /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/functional:1856:9
[MongoDFixture:job2]     #7 0x5564871bca7a in std::function<mongo::Status (mongo::InitializerContext*)>::operator()(mongo::InitializerContext*) const /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/functional:2267:14
[MongoDFixture:job2]     #8 0x5564871bc1c0 in mongo::Initializer::execute(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) const /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/base/initializer.cpp:57:22
[MongoDFixture:job2]     #9 0x5564871bc5e4 in mongo::runGlobalInitializers(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/base/initializer.cpp:70:35
[MongoDFixture:job2]     #10 0x5564871bc79e in mongo::runGlobalInitializers(int, char const* const*, char const* const*) /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/base/initializer.cpp:89:12
[MongoDFixture:job2]     #11 0x55648574f105 in mongoDbMain(int, char**, char**) /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/db/db.cpp:1074:21
[MongoDFixture:job2]     #12 0x55648574f105 in main /data/mci/986c5e304c8c21ee8a45596ddedbfbbe/src/src/mongo/db/db.cpp:779
[MongoDFixture:job2]     #13 0x7f88f4b4782f in __libc_start_main /build/glibc-9tT8Do/glibc-2.23/csu/../csu/libc-start.c:291
[MongoDFixture:job2]     #14 0x55648572fdc8 in _start (/data/mci/f8008ddfb24550f379e521aaaaa119a5/src/mongod+0x28bfdc8)
[MongoDFixture:job2] 
[MongoDFixture:job2] mongod on port 20500 was expected to be running in _do_teardown(), but wasn't. Exited with code 1.

But the task was not marked as a failure, instead thinking that no tests were run:

 [2017/06/07 16:09:03.485] [executor] 2017-06-07T16:09:03.484+0000 Shuffling order of tests for js_tests in suite aggregation. The seed is 383194046260.
 [2017/06/07 16:09:03.523] [executor:js_test] 2017-06-07T16:09:03.523+0000 Writing output of job #0 to https://logkeeper.mongodb.org/build/0662ee75067fe7d2d232f8ca185354ff.
 [2017/06/07 16:09:03.558] [executor:js_test] 2017-06-07T16:09:03.558+0000 Writing output of job #1 to https://logkeeper.mongodb.org/build/86b96d4afed95e2a0ffb66ebb7cd48f3.
 [2017/06/07 16:09:03.587] [executor:js_test] 2017-06-07T16:09:03.587+0000 Writing output of job #2 to https://logkeeper.mongodb.org/build/5430ad712810ffd63287eebd20b10e9a.
 [2017/06/07 16:09:03.617] [executor:js_test] 2017-06-07T16:09:03.617+0000 Writing output of job #3 to https://logkeeper.mongodb.org/build/3e456106bb72af96973a236912edc9e0.
 [2017/06/07 16:09:03.685] [executor:js_test] 2017-06-07T16:09:03.685+0000 Writing output of job #4 to https://logkeeper.mongodb.org/build/7f65119b4b8390edb91f734db12c6f6b.
 [2017/06/07 16:09:03.685] [executor:js_test] 2017-06-07T16:09:03.685+0000 Starting execution of js_tests...
 [2017/06/07 16:09:04.333] [executor:js_test] 2017-06-07T16:09:04.332+0000 Encountered an error while waiting for MongoDFixture (Job #0) to be ready
 [2017/06/07 16:09:04.333] Traceback (most recent call last):
 [2017/06/07 16:09:04.333]   File "/data/mci/b1773361f21e55b5703554055181ff2d/src/buildscripts/resmokelib/testing/executor.py", line 128, in _setup_fixtures
 [2017/06/07 16:09:04.333]     job.fixture.await_ready()
 [2017/06/07 16:09:04.333]   File "/data/mci/b1773361f21e55b5703554055181ff2d/src/buildscripts/resmokelib/testing/fixtures/standalone.py", line 100, in await_ready
 [2017/06/07 16:09:04.333]     " unexpectedly with code %d." % (self.port, exit_code))
 [2017/06/07 16:09:04.333] ServerFailure: Could not connect to mongod on port 20000, process ended unexpectedly with code 1.
 [2017/06/07 16:09:04.333] [executor:js_test] 2017-06-07T16:09:04.333+0000 Teardown of MongoDFixture (Job #0) was not successful.
 [2017/06/07 16:09:04.333] [executor:js_test] 2017-06-07T16:09:04.333+0000 Teardown of MongoDFixture (Job #1) was not successful.
 [2017/06/07 16:09:04.333] [executor:js_test] 2017-06-07T16:09:04.333+0000 Teardown of MongoDFixture (Job #2) was not successful.
 [2017/06/07 16:09:04.334] [executor:js_test] 2017-06-07T16:09:04.334+0000 Teardown of MongoDFixture (Job #3) was not successful.
 [2017/06/07 16:09:04.334] [executor:js_test] 2017-06-07T16:09:04.334+0000 Teardown of MongoDFixture (Job #4) was not successful.
 [2017/06/07 16:09:04.334] [resmoke] 2017-06-07T16:09:04.334+0000 ================================================================================
 [2017/06/07 16:09:04.334] [resmoke] 2017-06-07T16:09:04.334+0000 Summary of aggregation suite: No tests ran.
 [2017/06/07 16:09:04.334] Suite did not run any tests.
 [2017/06/07 16:09:04.948] [resmoke] 2017-06-07T16:09:04.947+0000 resmoke.py invocation: buildscripts/resmoke.py --suites=aggregation --storageEngine=mmapv1 --jobs=5 --shuffle --continueOnFailure --storageEngineCacheSizeGB=1 --log=buildlogger --staggerJobs=on --tagFile=etc/test_lifecycle.yml --taskId=mongodb_mongo_master_ubuntu1604_debug_ubsan_aggregation_10e3bc5992cb537a1a23722143aacc101431d95a_17_06_07_15_22_22 --includeWithAnyTags=unreliable,unreliable|aggregation,unreliable|aggregation|ubuntu1604-debug-ubsan,unreliable|aggregation|ubuntu1604-debug-ubsan|ubuntu1604-build --reportFailureStatus=silentfail --reportFile=report_unreliable.json
 [2017/06/07 16:09:04.968] [resmoke] 2017-06-07T16:09:04.968+0000 YAML configuration of suite aggregation
 [2017/06/07 16:09:04.968] test_kind: js_test
 [2017/06/07 16:09:04.968] selector:
 [2017/06/07 16:09:04.968]   roots:
 [2017/06/07 16:09:04.968]   - jstests/aggregation/*.js
 [2017/06/07 16:09:04.968]   - jstests/aggregation/bugs/*.js
 [2017/06/07 16:09:04.968]   - jstests/aggregation/expressions/*.js
 [2017/06/07 16:09:04.968]   - jstests/aggregation/sources/*/*.js
 [2017/06/07 16:09:04.968]   - jstests/aggregation/variables/*.js
 [2017/06/07 16:09:04.968] executor:
 [2017/06/07 16:09:04.968]   config:
 [2017/06/07 16:09:04.968]     shell_options:
 [2017/06/07 16:09:04.968]       readMode: commands
 [2017/06/07 16:09:04.968]   fixture:
 [2017/06/07 16:09:04.968]     class: MongoDFixture
 [2017/06/07 16:09:04.968]     mongod_options:
 [2017/06/07 16:09:04.968]       set_parameters:
 [2017/06/07 16:09:04.968]         enableTestCommands: 1
 [2017/06/07 16:09:04.968]   hooks:
 [2017/06/07 16:09:04.968]   - class: ValidateCollections
 [2017/06/07 16:09:04.968]   - class: CleanEveryN
 [2017/06/07 16:09:04.968]     n: 20
 [2017/06/07 16:09:04.968] logging:
 [2017/06/07 16:09:04.968]   executor:
 [2017/06/07 16:09:04.969]     format: '[%(name)s] %(asctime)s %(message)s'
 [2017/06/07 16:09:04.969]     handlers:
 [2017/06/07 16:09:04.969]     - class: logging.StreamHandler
 [2017/06/07 16:09:04.969]   fixture:
 [2017/06/07 16:09:04.969]     format: '[%(name)s] %(message)s'
 [2017/06/07 16:09:04.969]     handlers:
 [2017/06/07 16:09:04.969]     - class: buildlogger
 [2017/06/07 16:09:04.969]   tests:
 [2017/06/07 16:09:04.969]     format: '[%(name)s] %(asctime)s %(message)s'
 [2017/06/07 16:09:04.969]     handlers:
 [2017/06/07 16:09:04.969]     - class: buildlogger
 [2017/06/07 16:09:04.969] [executor] 2017-06-07T16:09:04.968+0000 Shuffling order of tests for js_tests in suite aggregation. The seed is 383194046701.
 [2017/06/07 16:09:04.969] [executor] 2017-06-07T16:09:04.968+0000 Skipping js_tests, no tests to run
 [2017/06/07 16:09:04.969] [resmoke] 2017-06-07T16:09:04.968+0000 ================================================================================
 [2017/06/07 16:09:04.969] [resmoke] 2017-06-07T16:09:04.968+0000 Summary of aggregation suite: No tests ran.
 [2017/06/07 16:09:04.969] Suite did not run any tests.
 [2017/06/07 16:09:05.151] Task completed - SUCCESS.



 Comments   
Comment by Githook User [ 09/Jun/17 ]

Author:

{u'username': u'elouie99', u'name': u'Eddie Louie', u'email': u'eddie.louie@mongodb.com'}

Message: SERVER-29507 Mark task failed if resmoke.py fails to start fixtures
Branch: master
https://github.com/mongodb/mongo/commit/6f06cf3ea127ce3706510b46c5bd0937ad7fba1b

Comment by Eddie Louie [ 08/Jun/17 ]

Code review: https://mongodbcr.appspot.com/137200002

Evergreen patch: https://evergreen.mongodb.com/version/593a1fff2fbabe689c005acc

Comment by Jonathan Abrahams [ 08/Jun/17 ]

We should modify the "run "test" function in etc/evergreen.yml to save the $? status generated from resmoke.py and exit with that combined status from the 2 invocations to resmoke.py.

Comment by Charlie Swanson [ 07/Jun/17 ]

cc jonathan.abrahams, marking as related to SERVER-28784 as Max requested.

Generated at Thu Feb 08 04:21:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.