[SERVER-10817] httpinterface.js failing on Nightly Linux 64-bit Created: 19/Sep/13  Updated: 11/Jul/16  Resolved: 20/Sep/13

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 2.5.3

Type: Bug Priority: Major - P3
Reporter: Matt Kangas Assignee: Matt Dannenberg
Resolution: Done Votes: 0
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

I am able to occasionally reproduce this on a Ubuntu 12.04 box (my own build box):

./buildscripts/smoke.py --mode=files jstests/slowNightly/httpinterface.js

Participants:

 Description   

This has been failing since Sep 10! Nightly Linux 64-bit (last 50 builds)

Build #1576 rev 5532f53f (Wed Sep 11) through
Build #1585 rev 166e0012 (Wed Sep 18)

smokeJsSlowNightly : jstests/slowNightly/httpinterface.js

 *******************************************
         Test : httpinterface.js ...
      Command : /mntfast/data/slave/Linux_64bit_Nightly/mongo/mongo --port 27999 --authenticationMechanism MONGODB-CR /mntfast/data/slave/Linux_64bit_Nightly/mongo/jstests/slowNightly/httpinterface.js --eval TestData = new Object();TestData.testPath = "/mntfast/data/slave/Linux_64bit_Nightly/mongo/jstests/slowNightly/httpinterface.js";TestData.testFile = "httpinterface.js";TestData.testName = "httpinterface";TestData.setParameters = "";TestData.noJournal = false;TestData.noJournalPrealloc = false;TestData.auth = false;TestData.keyFile = null;TestData.keyFileData = null;TestData.authMechanism = "MONGODB-CR";
         Date : Wed Sep 18 00:57:45 2013
                (output suppressed; see http://buildlogs.mongodb.org/build/523932c8d2a60f175b000e25/test/523932c9d2a60f175b000e33/)
                 2938.8599 ms

http://buildlogs.mongodb.org/Nightly%20Linux%2064-bit/builds/1585/test/recent%20failures/httpinterface.js

2013-09-18T00:57:46.940-0400 shell: started program /mntfast/data/slave/Linux_64bit_Nightly/mongo/mongod --port 31000 --smallfiles --httpinterface --dbpath /data/db/mongod-31000 --setParameter enableTestCommands=1
 m31000| 2013-09-18T00:57:46.987-0400 [initandlisten] MongoDB starting : pid=11098 port=31000 dbpath=/data/db/mongod-31000 64-bit host=ip-10-2-29-40
 m31000| 2013-09-18T00:57:46.987-0400 [initandlisten] 
 m31000| 2013-09-18T00:57:46.987-0400 [initandlisten] ** NOTE: This is a development version (2.5.3-pre-) of MongoDB.
 m31000| 2013-09-18T00:57:46.987-0400 [initandlisten] **       Not recommended for production.
 m31000| 2013-09-18T00:57:46.987-0400 [initandlisten] 
 m31000| 2013-09-18T00:57:46.987-0400 [initandlisten] db version v2.5.3-pre-
 m31000| 2013-09-18T00:57:46.987-0400 [initandlisten] git version: 166e00122ccd947beddae28274f38677cc789980
 m31000| 2013-09-18T00:57:46.987-0400 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
 m31000| 2013-09-18T00:57:46.987-0400 [initandlisten] allocator: tcmalloc
 m31000| 2013-09-18T00:57:46.987-0400 [initandlisten] options: { dbpath: "/data/db/mongod-31000", httpinterface: true, port: 31000, setParameter: [ "enableTestCommands=1" ], smallfiles: true }
 m31000| 2013-09-18T00:57:46.990-0400 [initandlisten] journal dir=/data/db/mongod-31000/journal
 m31000| 2013-09-18T00:57:46.991-0400 [initandlisten] recover : no journal files present, no recovery needed
 m31000| 2013-09-18T00:57:47.110-0400 [FileAllocator] allocating new datafile /data/db/mongod-31000/local.0, filling with zeroes...
 m31000| 2013-09-18T00:57:47.110-0400 [FileAllocator] creating directory /data/db/mongod-31000/_tmp
 m31000| 2013-09-18T00:57:47.136-0400 [FileAllocator] done allocating datafile /data/db/mongod-31000/local.0, size: 16MB,  took 0.024 secs
 m31000| 2013-09-18T00:57:47.137-0400 [FileAllocator] allocating new datafile /data/db/mongod-31000/local.ns, filling with zeroes...
2013-09-18T00:57:47.143-0400 Error: couldn't connect to server localhost:32000 at /mntfast/data/slave/Linux_64bit_Nightly/mongo/jstests/slowNightly/httpinterface.js:12
failed to load: /mntfast/data/slave/Linux_64bit_Nightly/mongo/jstests/slowNightly/httpinterface.js

Note that neither of these messages are visible above:

 m31000| ... [initandlisten] waiting for connections on port 31000
 m31000| ... [websvr] admin web console waiting for connections on port 32000

Did we introduce something that causes the server to start more slowly, and perhaps there's a race if the test doesn't wait? Or is it something else?

The last green build on Nightly Linux 64-bit (buildbot) was #1575. Commits between this and the first failure:

git log 157e4342c22c..5532f53fe8eb

5532f53 2013-09-10 17:25 matt dannenberg    o SERVER-10724 better exception handling in health poll
2c1a91e 2013-09-10 10:03 Randolph Tan       o SERVER-10688 sync6.js - verify failed in DistributedLock:lock_try
ebd13ab 2013-09-06 15:19 matt dannenberg    o SERVER-8375 sync from primary on clock skew
ce6607f 2013-08-29 12:08 Hari Khalsa        o SERVER-10285 uassert that s2latlng is valid, print out if it's not
894828f 2013-09-09 16:54 matt dannenberg    o SERVER-10707 use ScopeGuard to close sockets in setupSockets
e284002 2013-09-04 14:47 matt dannenberg    o SERVER-9818 allocate socket file descriptors before opening datafiles



 Comments   
Comment by Matt Dannenberg [ 20/Sep/13 ]

fixed by removing a race condition via the following commit:
https://github.com/mongodb/mongo/commit/9bb04443ccbcdd4f462cb2f07532c977a39bbf0d

Generated at Thu Feb 08 03:24:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.