Critical - P2 An automated testing patch build recently failed with the following assertion:
[conn15] WiredTiger error (0) [1491418210:100400][5596:2001941376], file:collection-1413-2830616528214231890.wt, WT_SESSION.verify: src\third_party\wiredtiger\src\evict\evict_file.c, 100: F_ISSET(session->dhandle, WT_DHANDLE_DEAD) || __wt_page_can_evict(session, ref, NULL) [conn15] WiredTiger error (0) [1491418210:101400][5596:2001941376], file:collection-1413-2830616528214231890.wt, WT_SESSION.verify: aborting WiredTiger library
The call stack of the failing thread is:
[conn15] mongod.exe ...\src\mongo\util\stacktrace_windows.cpp(239) mongo::printStackTrace+0x43 [conn15] mongod.exe ...\src\mongo\util\signal_handlers_synchronous.cpp(180) mongo::`anonymous namespace'::printSignalAndBacktrace+0x73 [conn15] mongod.exe ...\src\mongo\util\signal_handlers_synchronous.cpp(236) mongo::`anonymous namespace'::abruptQuit+0x83 [conn15] mongod.exe d:\th\minkernel\crts\ucrt\src\appcrt\misc\signal.cpp(522) raise+0x468 [conn15] mongod.exe d:\th\minkernel\crts\ucrt\src\appcrt\startup\abort.cpp(71) abort+0x39 [conn15] mongod.exe ...\src\third_party\wiredtiger\src\os_common\os_abort.c(25) __wt_abort+0x22 [conn15] mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(486) __wt_assert+0x37 [conn15] mongod.exe ...\src\third_party\wiredtiger\src\evict\evict_file.c(101) __wt_evict_file+0x1a7 [conn15] mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(203) __wt_conn_btree_sync_and_close+0xd3 [conn15] mongod.exe ...\src\third_party\wiredtiger\src\conn\conn_dhandle.c(524) __wt_conn_dhandle_close_all+0x17d [conn15] mongod.exe ...\src\third_party\wiredtiger\src\schema\schema_worker.c(52) __wt_schema_worker+0x110 [conn15] mongod.exe ...\src\third_party\wiredtiger\src\schema\schema_worker.c(96) __wt_schema_worker+0x3ea [conn15] mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(1377) __session_verify+0x298 [conn15] mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp(439) mongo::WiredTigerUtil::verifyTable+0x1fa [conn15] mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_record_store.cpp(1548) mongo::WiredTigerRecordStore::validate+0x9b [conn15] mongod.exe ...\src\mongo\db\catalog\collection.cpp(1303) mongo::Collection::validate+0x7bc [conn15] mongod.exe ...\src\mongo\db\commands\validate.cpp(128) mongo::ValidateCmd::run+0x55b [conn15] mongod.exe ...\src\mongo\db\commands\dbcommands.cpp(1450) mongo::Command::run+0x6c9 [conn15] mongod.exe ...\src\mongo\db\commands\dbcommands.cpp(1693) mongo::execCommandDatabase+0x10b4 [conn15] mongod.exe ...\src\mongo\db\commands.cpp(453) mongo::Command::execCommand+0x38 [conn15] mongod.exe ...\src\mongo\db\run_commands.cpp(73) mongo::runCommands+0x4ba [conn15] mongod.exe ...\src\mongo\db\assemble_response.cpp(222) mongo::`anonymous namespace'::receivedRpc+0x190 [conn15] mongod.exe ...\src\mongo\db\assemble_response.cpp(558) mongo::assembleResponse+0x64b [conn15] mongod.exe ...\src\mongo\db\service_entry_point_mongod.cpp(139) mongo::ServiceEntryPointMongod::_sessionLoop+0x180 [conn15] mongod.exe c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional(212) std::_Func_impl<<lambda_0476eb5123845e42a2f452f2efde6866>,std::allocator<int>,void,std::shared_ptr<mongo::transport::Session> const & __ptr64>::_Do_call+0x43 [conn15] mongod.exe ...\src\mongo\transport\service_entry_point_utils.cpp(78) mongo::`anonymous namespace'::runFunc+0x1a6 [conn15] mongod.exe c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(247) std::_LaunchPad<std::unique_ptr<std::tuple<std::_Binder<std::_Unforced,void * __ptr64 (__cdecl&)(void * __ptr64),mongo::`anonymous namespace'::Context * __ptr64> >,std::default_delete<std::tuple<std::_Binder<std::_Unforced,void * __ptr64 (__cdecl&)(void * __ptr64),mongo::`anonymous namespace'::Context * __ptr64> > > > >::_Run+0x6f [conn15] mongod.exe c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(210) std::_Pad::_Call_func+0x9 [conn15] mongod.exe d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp(92) invoke_thread_procedure+0x2c [conn15] mongod.exe d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp(115) thread_start<unsigned int (__cdecl*)(void * __ptr64)>+0x93 [conn15] kernel32.dll BaseThreadInitThunk+0xd
The test was run including the changes in WT-3193, in fact all changes in WiredTiger up to here
The test was run on a Windows 2008R2 machine using a debug build.
The command used to run the test was equivalent to:
python buildscripts/resmoke.py --suites=concurrency_sharded --storageEngine=wiredTiger --shuffle --continueOnFailure --staggerJobs=on