[SERVER-14870] smoke.py should show mongod startup logs if mongod startup fails Created: 12/Aug/14  Updated: 14/Apr/16  Resolved: 08/May/15

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Michael O'Brien Assignee: Max Hirschhorn
Resolution: Duplicate Votes: 0
Labels: community-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-1424 smoke.py could use some work Closed
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Quint Iteration 3
Participants:
Linked BF Score: 0

 Description   

If mongod can't start up successfully for some reason, smoke.py will just complain "timed out starting mongod" and exit. However, smoke.py also swallows all the logging output from the mongod, so there's no way of seeing why it failed to start up (could be caused by a crash, machine environment, invalid command line args, etc.).

smoke.py should at the very least show the mongod's output so that the reason for the startup failure can be debugged.



 Comments   
Comment by Max Hirschhorn [ 04/May/15 ]

This seems more like a buildlogger issue that will be resolved after switching over to logkeeper than a smoke.py issue.

Running python buildscripts/smoke.py jsCore --storageEngine=fakeStorageEngine and logging to stdout, the error message is visible in the output.

[Errno 111] Connection refused
clean_dbroot: /data/db/sconsTests/
num procs:342
running /home/maxh/debugging/mongo/mongod --port 27999 --dbpath /data/db/sconsTests/ --setParameter enableTestCommands=1 --httpinterface --storageEngine fakeStorageEngine
2015-05-04T08:46:02.962-0400 I STORAGE  [initandlisten] exception in initAndListen: 18656 Cannot start server with an unknown storage engine: fakeStorageEngine, terminating
2015-05-04T08:46:02.963-0400 I CONTROL  [initandlisten] dbexit:  rc: 100

Similarly for resmoke.py.

python buildscripts/resmoke.py --suites=core --storageEngine=fakeStorageEngine
 
2015-05-04 08:34:49,388 [executor:js_test] Starting execution of js_tests...
[MongoDFixture:job0] Starting MongoDB server on port 34937...
./mongod --setParameter enableTestCommands=1 --storageEngine fakeStorageEngine --nopreallocj --dbpath /data/db/job0/resmoke --port 34937
[MongoDFixture:job0] MongoDB server started on port 34937 with pid 25526.
[MongoDFixture:job0] Waiting to connect to MongoDB server on port 34937.
[MongoDFixture:job0] 2015-05-04T08:34:49.478-0400 I STORAGE  [initandlisten] exception in initAndListen: 18656 Cannot start server with an unknown storage engine: fakeStorageEngine, terminating
[MongoDFixture:job0] 2015-05-04T08:34:49.478-0400 I CONTROL  [initandlisten] dbexit:  rc: 100
2015-05-04 08:34:50,398 [executor:js_test] Encountered an error while waiting for MongoDFixture (Job #0) to be ready
Traceback (most recent call last):
  File "/home/maxh/debugging/mongo/buildscripts/resmokelib/testing/executor.py", line 118, in _setup_fixtures
    job.fixture.await_ready()
  File "/home/maxh/debugging/mongo/buildscripts/resmokelib/testing/fixtures/standalone.py", line 98, in await_ready
    raise errors.ServerFailure("Could not connect to server, process ended"
ServerFailure: Could not connect to server, process ended unexpectedly.

The main problem is that the buildlogger server does not provide a way to view the global logs outside the time range of a test's execution. Another issue is that smoke.py only logs the buildlogger test endpoint (/build/<build_id>/test/<test_id>) when a test is being run. The latter is resolved by resmoke.py because the buildlogger build endpoint (/build/<build_id>) is logged when starting up the fixture. Thus, the global logs will always be accessible via resmoke.py + logkeeper.

Generated at Thu Feb 08 03:36:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.