[SERVER-10816] MCI: all test suites failing on windows-64-2k8-debug, Windows 64-bit DEBUG Created: 19/Sep/13  Updated: 11/Jul/16  Resolved: 25/Sep/13

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 2.5.3

Type: Bug Priority: Major - P3
Reporter: Matt Kangas Assignee: Matt Dannenberg
Resolution: Done Votes: 0
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

MCI windows-64-2k8-debug
MCI Windows 64-bit DEBUG


Issue Links:
Related
is related to SERVER-10726 memory.js taking 7+ hours on Linux 64... Closed
Operating System: ALL
Participants:

 Description   

Drew, is this related to these recent changes of yours? MCI seems to think so.

681bc4c 09/18/13 21:40pm Andrew Morrow
be4bc7d 09/18/13 21:40pm Andrew Morrow

Everything gone boom here:

MCI 681bc4c9be6a... Windows 64-bit DEBUG
MCI 681bc4c9be6a... Windows 64-bit 2008R2+ DEBUG

Sample failure messages: (there are many more...)

http://buildlogs.mongodb.org/mci_0.9_windows_64_2k8_debug/builds/249/test/core_0/test.exe

2013-09-19T02:02:19.086+0000 [testsuite] going to run suite: count
...
2013-09-19T02:02:19.099+0000 [testsuite] 	 going to run test: class CountTests::Yield
2013-09-19T02:02:19.099+0000 [testsuite] build index on: unittests.counttests properties: { "v" : 1, "key" : { "a" : 1 }, "ns" : "unittests.counttests", "name" : "a" }
2013-09-19T02:02:19.099+0000 [testsuite] build index done.  scanned 0 total records. 0 secs
2013-09-19T02:02:19.113+0000 [testsuite] debug nsincecommitifneeded:80 bytes:876544
2013-09-19T02:02:19.120+0000 [testsuite] debug nsincecommitifneeded:120 bytes:1040384
2013-09-19T02:02:19.125+0000 [testsuite] debug nsincecommitifneeded:160 bytes:1204224
2013-09-19T02:02:19.132+0000 [testsuite] debug nsincecommitifneeded:200 bytes:1368064
2013-09-19T02:02:19.140+0000 [testsuite] debug nsincecommitifneeded:240 bytes:1552384
2013-09-19T02:02:19.140+0000 [testsuite] _DEBUG printing stack given high nsinccommitifneeded number
2013-09-19T02:02:19.167+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.167+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.167+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.167+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.167+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.167+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.167+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.167+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.168+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.168+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.168+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.168+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.168+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.168+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.168+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.168+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.168+0000 [testsuite] test.exe        ???
2013-09-19T02:02:19.168+0000 [testsuite] kernel32.dll    BaseThreadInitThunk+0xd
2013-09-19T02:02:19.168+0000 [testsuite] 
2013-09-19T02:02:19.175+0000 [testsuite] debug nsincecommitifneeded:280 bytes:1716224
2013-09-19T02:02:19.182+0000 [testsuite] debug nsincecommitifneeded:320 bytes:1880064
2013-09-19T02:02:19.187+0000 [testsuite] debug nsincecommitifneeded:360 bytes:2043904
2013-09-19T02:02:19.195+0000 [testsuite] debug nsincecommitifneeded:400 bytes:2207744
2013-09-19T02:02:19.201+0000 [testsuite] debug nsincecommitifneeded:440 bytes:2379776
2013-09-19T02:02:19.207+0000 [testsuite] debug nsincecommitifneeded:480 bytes:2543616
2013-09-19T02:02:19.213+0000 [testsuite] debug nsincecommitifneeded:520 bytes:2715648
2013-09-19T02:02:19.220+0000 [testsuite] debug nsincecommitifneeded:560 bytes:2879488
2013-09-19T02:02:19.227+0000 [testsuite] debug nsincecommitifneeded:600 bytes:3043328
2013-09-19T02:02:19.232+0000 [testsuite] debug nsincecommitifneeded:640 bytes:3211264
2013-09-19T02:02:19.238+0000 [testsuite] debug nsincecommitifneeded:680 bytes:3383296
2013-09-19T02:02:19.245+0000 [testsuite] debug nsincecommitifneeded:720 bytes:3547136
2013-09-19T02:02:19.251+0000 [testsuite] debug nsincecommitifneeded:760 bytes:3723264
2013-09-19T02:02:19.257+0000 [testsuite] debug nsincecommitifneeded:800 bytes:3887104
2013-09-19T02:02:19.265+0000 [dummy writer] thread dummy writer stack usage was 337920 bytes,  which is the most so far
2013-09-19T02:02:19.265+0000 [dummy writer] used 337920 bytes, max is 337920 exiting
2013-09-19T02:02:19.265+0000 [dummy writer] Fatal Assertion 16151
2013-09-19T02:02:19.289+0000 [dummy writer] test.exe        ???
2013-09-19T02:02:19.289+0000 [dummy writer] test.exe        ???
2013-09-19T02:02:19.289+0000 [dummy writer] test.exe        ???
2013-09-19T02:02:19.289+0000 [dummy writer] test.exe        ???
2013-09-19T02:02:19.289+0000 [dummy writer] test.exe        ???
2013-09-19T02:02:19.289+0000 [dummy writer] test.exe        ???
2013-09-19T02:02:19.289+0000 [dummy writer] test.exe        ???
2013-09-19T02:02:19.289+0000 [dummy writer] test.exe        ???
2013-09-19T02:02:19.289+0000 [dummy writer] test.exe        ???
2013-09-19T02:02:19.289+0000 [dummy writer] kernel32.dll    BaseThreadInitThunk+0xd

http://buildlogs.mongodb.org/mci_0.9_windows_64_2k8_debug/builds/249/test/aggregation_0/mongos_slaveok.js

2013-09-19T02:04:30.870+0000 [IndexRebuilder] mongod.exe      ...\src\mongo\util\assert_util.cpp(132)                          mongo::fassertFailed+0xb7
2013-09-19T02:04:30.870+0000 [IndexRebuilder] mongod.exe      ...\src\mongo\db\client.cpp(126)                                 mongo::StackChecker::check+0x2cb
2013-09-19T02:04:30.870+0000 [IndexRebuilder] mongod.exe      ...\src\mongo\db\client.cpp(240)                                 mongo::Client::shutdown+0x51
2013-09-19T02:04:30.870+0000 [IndexRebuilder] mongod.exe      ...\src\mongo\util\scopeguard.h(115)                             mongo::ScopeGuardImplBase::SafeExecute<mongo::ObjScopeGuardImpl0<mongo::Client,bool (__cdecl mongo::Client::*)(void) __ptr64> >+0x20
2013-09-19T02:04:30.870+0000 [IndexRebuilder] mongod.exe      ...\src\mongo\db\index_rebuilder.cpp(88)                         mongo::IndexRebuilder::run+0x15e5
2013-09-19T02:04:30.870+0000 [IndexRebuilder] mongod.exe      ...\src\mongo\util\background.cpp(68)                            mongo::BackgroundJob::jobBody+0x3f0
2013-09-19T02:04:30.871+0000 [IndexRebuilder] mongod.exe      ...\src\third_party\boost\boost\bind\mem_fn_template.hpp(165)    boost::_mfi::mf1<void,mongo::BackgroundJob,boost::shared_ptr<mongo::BackgroundJob::JobStatus> >::operator()+0x47
2013-09-19T02:04:30.871+0000 [IndexRebuilder] mongod.exe      ...\src\third_party\boost\boost\thread\detail\thread.hpp(63)     boost::detail::thread_data<boost::_bi::bind_t<void,boost::_mfi::mf1<void,mongo::BackgroundJob,boost::shared_ptr<mongo::BackgroundJob::JobStatus> >,boost::_bi::list2<boost::_bi::value<mongo::BackgroundJob * __ptr64>,boost::_bi::value<boost::shared_ptr<mongo::BackgroundJob::JobStatus> > > > >::run+0x31
2013-09-19T02:04:30.871+0000 [IndexRebuilder] mongod.exe      ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)  boost::`anonymous namespace'::thread_start_function+0x21
2013-09-19T02:04:30.871+0000 [IndexRebuilder] mongod.exe      f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)      _callthreadstartex+0x25
2013-09-19T02:04:30.871+0000 [IndexRebuilder] mongod.exe      f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(297)      _threadstartex+0xb4
2013-09-19T02:04:30.871+0000 [IndexRebuilder] kernel32.dll                                                                     BaseThreadInitThunk+0xd
2013-09-19T02:04:30.871+0000 [IndexRebuilder] 
2013-09-19T02:04:30.871+0000 [IndexRebuilder] writing minidump diagnostic file mongo.dmp
2013-09-19T02:04:30.877+0000 [FileAllocator] done allocating datafile /data/db/sconsTests\local.0, size: 64MB,  took 2.069 secs
2013-09-19T02:04:30.877+0000 [initandlisten] datafileheader::init initializing /data/db/sconsTests\local.0 n:0
2013-09-19T02:04:30.878+0000 [FileAllocator] allocating new datafile /data/db/sconsTests\local.ns, filling with zeroes...
2013-09-19T02:04:30.933+0000 [IndexRebuilder] *** immediate exit due to unhandled exception



 Comments   
Comment by auto [ 25/Sep/13 ]

Author:

{u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: SERVER-10816 enable StackChecker when not optimized and in debug mode rather than when simply in debug mode
Branch: master
https://github.com/mongodb/mongo/commit/97c82b16b21d2fe06f933e406909322115e79eaf

Comment by Andy Schwerin [ 24/Sep/13 ]

The stack checker and related utilities depend on compiler optimizations being disabled, so should only be enabled when --dbg=on and --opt=off.

Comment by Matt Dannenberg [ 24/Sep/13 ]

It is related. The planned fix should fix that as well.

Comment by Matt Kangas [ 24/Sep/13 ]

Related? core suite is failing on Linux-64-debug. Earliest occurrence is here:

http://mci.10gen.com/ui/task/mongodb_mongo_master_linux_64_debug_73ca6bf23e0d37ee781085466df6c989558c64a3_13_09_19_05_22_08_core_linux_64
http://buildlogs.mongodb.org/mci_0.9_linux_64_debug/builds/868/test/core_0/test

2013-09-19T08:17:21.778+0000 [testsuite] going to run suite: stack
2013-09-19T08:17:21.778+0000 [testsuite] 	 going to run test: StackTests::InCons
2013-09-19T08:17:21.782+0000 [testsuite] FAIL: StackTests::InCons	Expected: b.a @src/mongo/dbtests/stacktests.cpp:75

Comment by Andrew Morrow (Inactive) [ 19/Sep/13 ]

Yeah, looks like windows debug builds with optimization has issues. Will take a closer look tomorrow.

Generated at Thu Feb 08 03:24:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.