[SERVER-23588] mongod with WiredTiger won't start on Windows when built with --dbg=on --opt=off Created: 06/Apr/16  Updated: 22/Nov/16  Resolved: 08/Apr/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.3.5
Fix Version/s: 3.2.6, 3.3.5

Type: Bug Priority: Major - P3
Reporter: Kaloian Manassiev Assignee: Alexander Gorrod
Resolution: Done Votes: 0
Labels: code-only
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to WT-2540 Separate stream and file handle methods Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

Build mongod.exe with the following command line:

scons -j 12 --dbg=on --opt=off --dynamic-windows --win-version-min=ws08r2 mongod.exe

Participants:

 Description   

The WT bootstrap code double-opens the .turtle file, which causes validation exception and process termination in Windows debug builds. The second open happens at this line and the file has previously been opened here.

With this problem debug builds on Windows are unusable and evergreen does not catch it because we build with --dbg=on --opt=off, which includes the release runtime.

This is the call stack:

>	msvcr120.dll!_invoke_watson(const wchar_t * pszExpression=0x000001d20000000b, const wchar_t * pszFunction=0x00007ffcb1d67adf, const wchar_t * pszFile=0x0000001d00000000, unsigned int nLine=1722168464, unsigned __int64 pReserved=0) Line 132	C++
 	msvcr120.dll!_invalid_parameter(const wchar_t * pszExpression=0x0000000000000000, const wchar_t * pszFunction=0x0000000000000000, const wchar_t * pszFile=0x0000000000000000, unsigned int nLine=0, unsigned __int64 pReserved=0) Line 85	C++
 	msvcr120.dll!_invalid_parameter_noinfo() Line 97	C++
 	msvcr120.dll!_fdopen(int filedes, const char * mode) Line 55	C
 	mongod.exe!__win_handle_open(__wt_session_impl * session=0x000001d2f3760000, __wt_fh * fh=0x000001d2f333f100, const char * name=0x000001d2f333d270, unsigned int file_type=16, unsigned int flags=72) Line 634	C
 	mongod.exe!__wt_open(__wt_session_impl * session=0x000001d2f3760000, const char * name=0x00007ff755c0edb8, unsigned int file_type=16, unsigned int flags=72, __wt_fh * * fhp=0x0000001d9373c9b8) Line 204	C
 	mongod.exe!__wt_turtle_read(__wt_session_impl * session=0x000001d2f3760000, const char * key=0x000001d2f33411c0, char * * valuep=0x0000001d9373cbc8) Line 261	C
 	mongod.exe!__wt_metadata_search(__wt_session_impl * session=0x000001d2f3760000, const char * key=0x000001d2f33411c0, char * * valuep=0x0000001d9373cbc8) Line 275	C
 	mongod.exe!__conn_btree_config_set(__wt_session_impl * session=0x000001d2f3760000) Line 250	C
 	mongod.exe!__wt_conn_btree_open(__wt_session_impl * session=0x000001d2f3760000, const char * * cfg=0x0000001d9373cfc8, unsigned int flags=0) Line 318	C
 	mongod.exe!__wt_session_get_btree(__wt_session_impl * session=0x000001d2f3760000, const char * uri=0x00007ff755c0aba0, const char * checkpoint=0x0000000000000000, const char * * cfg=0x0000001d9373cfc8, unsigned int flags=0) Line 529	C
 	mongod.exe!__wt_session_get_btree(__wt_session_impl * session=0x000001d2f3760000, const char * uri=0x00007ff755c0aba0, const char * checkpoint=0x0000000000000000, const char * * cfg=0x0000001d9373cfc8, unsigned int flags=0) Line 522	C
 	mongod.exe!__wt_session_get_btree_ckpt(__wt_session_impl * session=0x000001d2f3760000, const char * uri=0x00007ff755c0aba0, const char * * cfg=0x0000001d9373cfc8, unsigned int flags=0) Line 339	C
 	mongod.exe!__wt_curfile_open(__wt_session_impl * session=0x000001d2f3760000, const char * uri=0x00007ff755c0aba0, __wt_cursor * owner=0x0000000000000000, const char * * cfg=0x0000001d9373cfc8, __wt_cursor * * cursorp=0x0000001d9373d048) Line 553	C
 	mongod.exe!__session_open_cursor_int(__wt_session_impl * session=0x000001d2f3760000, const char * uri=0x00007ff755c0aba0, __wt_cursor * owner=0x0000000000000000, __wt_cursor * other=0x0000000000000000, const char * * cfg=0x0000001d9373cfc8, __wt_cursor * * cursorp=0x0000001d9373d048) Line 329	C
 	mongod.exe!__wt_open_cursor(__wt_session_impl * session=0x000001d2f3760000, const char * uri=0x00007ff755c0aba0, __wt_cursor * owner=0x0000000000000000, const char * * cfg=0x0000001d9373cfc8, __wt_cursor * * cursorp=0x0000001d9373d048) Line 383	C
 	mongod.exe!__wt_metadata_cursor_open(__wt_session_impl * session=0x000001d2f3760000, const char * config=0x0000000000000000, __wt_cursor * * cursorp=0x0000001d9373d048) Line 59	C
 	mongod.exe!__wt_metadata_cursor(__wt_session_impl * session=0x000001d2f3760000, __wt_cursor * * cursorp=0x0000000000000000) Line 103	C
 	mongod.exe!wiredtiger_open(const char * home=0x00007ff755d4b158, __wt_event_handler * event_handler=0x000001d2f26ad260, const char * config=0x000001d2f2518740, __wt_connection * * wt_connp=0x000001d2f26ad258) Line 2238	C
 	mongod.exe!mongo::WiredTigerKVEngine::WiredTigerKVEngine(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & canonicalName={...}, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & path={...}, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & extraOpenOptions={...}, unsigned __int64 cacheSizeGB=18, bool durable=true, bool ephemeral=false, bool repair=false, bool readOnly=false) Line 248	C++
 	mongod.exe!mongo::`anonymous namespace'::WiredTigerFactory::create(const mongo::StorageGlobalParams & params={...}, const mongo::StorageEngineLockFile * lockFile=0x000001d2f26b3070) Line 86	C++
 	mongod.exe!mongo::ServiceContextMongoD::initializeGlobalStorageEngine() Line 187	C++
 	mongod.exe!mongo::_initAndListen(int listenPort=27017) Line 568	C++
 	mongod.exe!mongo::initAndListen(int listenPort=27017) Line 757	C++
 	mongod.exe!mongoDbMain(int argc=3, char * * argv=0x000001d2f2501840, char * * envp=0x000001d2f2505e10) Line 1061	C++
 	mongod.exe!wmain(int argc=3, wchar_t * * argvW=0x000001d2f0abe7a0, wchar_t * * envpW=0x000001d2f0ad78c0) Line 795	C++

This is the error:

Unhandled exception at 0x00007FFC66AADD27 (msvcr120.dll) in mongod.exe: An invalid parameter was passed to a function that considers invalid parameters fatal.



 Comments   
Comment by Githook User [ 14/Apr/16 ]

Author:

{u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

Message: Import wiredtiger-wiredtiger-2.8.0-201-g7ea2631.tar.gz from wiredtiger branch mongodb-3.2

ref: 43e885a..7ea2631

SERVER-23504 Coverity analysis defect 98177: Resource leak
SERVER-23526 Replication relies on storage engines reporting a non-zero size for correctness
SERVER-23588 mongod with WiredTiger won't start on Windows when built with --dbg=on --opt=off
SERVER-23682 WiredTiger changes for MongoDB 3.2.6
WT-2330 in-memory configurations should not create on-disk collection files
WT-2507 Add upgrading documentation in preparation for 2.8 release.
WT-2512 wtperf: MSVC complains about float conversion in throttle code
WT-2513 conversion from 'int64_t' to 'uint32_t'
WT-2517 wtperf uses setvbuf in a way that isn't supported on Windows
WT-2522 Incorrect format code in message
WT-2525 in-memory configurations: miscellaneous cleanups
WT-2527 OS X compile error, missing POSIX_FADV_WILLNEED #define
WT-2528 style error in WiredTiger build
WT-2529 The readonly test case is crashing with a stack overflow
WT-2531 in-memory tables are allocating unnecessary memory
WT-2532 WT_STREAM_APPEND and WT_STREAM_LINE_BUFFER flag overlap
WT-2533 Ensure that in-memory tables don't report a zero size
WT-2534 Invalid transaction snapshots on PowerPC
Branch: v3.2
https://github.com/mongodb/mongo/commit/7ee4e4e493c3785fea489ee3508ca18526709c16

Comment by Githook User [ 08/Apr/16 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: Merge pull request #2638 from wiredtiger/server-23588-win-fopen

SERVER-23588 Stop using _open_osfhandle on Windows.
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/5206656dfbb725a031be35203fc05f3f3beedc6f

Comment by Githook User [ 08/Apr/16 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

Message: SERVER-23588 Stop using _open_osfhandle on Windows.

It introduced some odd failures on debug MongoDB builds. Switch
to using fopen instead.
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/e0fc972c47872ad517cd44c9c517eba01242bf8e

Comment by Githook User [ 08/Apr/16 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

Message: Import wiredtiger-wiredtiger-2.8.0-140-g7bcf6fc.tar.gz from wiredtiger branch mongodb-3.4

ref: 94c171d..7bcf6fc

SERVER-23588 mongod with WiredTiger won't start on Windows when built with --dbg=on --opt=off
SERVER-23517 WiredTiger changes for MongoDB 3.3.5
Branch: master
https://github.com/mongodb/mongo/commit/58fe2bfb350939a4b8c6e8ddd9735a49dc8d2db3

Comment by Githook User [ 08/Apr/16 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: Merge pull request #2638 from wiredtiger/server-23588-win-fopen

SERVER-23588 Stop using _open_osfhandle on Windows.
Branch: mongodb-3.4
https://github.com/wiredtiger/wiredtiger/commit/5206656dfbb725a031be35203fc05f3f3beedc6f

Comment by Githook User [ 08/Apr/16 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

Message: SERVER-23588 Stop using _open_osfhandle on Windows.

It introduced some odd failures on debug MongoDB builds. Switch
to using fopen instead.
Branch: mongodb-3.4
https://github.com/wiredtiger/wiredtiger/commit/e0fc972c47872ad517cd44c9c517eba01242bf8e

Comment by Githook User [ 08/Apr/16 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: Merge pull request #2638 from wiredtiger/server-23588-win-fopen

SERVER-23588 Stop using _open_osfhandle on Windows.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/5206656dfbb725a031be35203fc05f3f3beedc6f

Comment by Githook User [ 08/Apr/16 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

Message: SERVER-23588 Stop using _open_osfhandle on Windows.

It introduced some odd failures on debug MongoDB builds. Switch
to using fopen instead.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/e0fc972c47872ad517cd44c9c517eba01242bf8e

Comment by Alexander Gorrod [ 08/Apr/16 ]

Thanks kaloian.manassiev I was able to reproduce when I included the enterprise modules. I didn't have much luck figuring out where we were going wrong with the API usage exactly, but I ended up switching out our usage of _os_openfhandle and fdopen to using fopen instead. With that change, our code became simpler and the problem was resolved.

I've pushed that change into a pull request (https://github.com/wiredtiger/wiredtiger/pull/2638).

Comment by Kaloian Manassiev [ 07/Apr/16 ]

Thanks, Alex. I looked into this a little bit more and I noticed that it only happens when enterprise mongod.exe is built with --dbg=on --opt=off, but does not happen when only building the free version. This is the command line I used, which should apply for spawnhost:

scons -j 2 --dbg=on --opt=off --ssl --dynamic-windows --win-version-min=ws08r2 CPPPATH="c:/openssl/include c:/sasl/include c:/snmp/include" LIBPATH="c:/openssl/lib c:/sasl/lib c:/snmp/lib" --mute mongod.exe

As far as msvcr120.dll is involved, I am not sure why this gets linked when using these parameters. mark.benvenuto, do you have any idea?

Comment by Alexander Gorrod [ 07/Apr/16 ]

Another thing I noticed is that the stack trace you posted is using msvcr120.dll, but you mention building against the debug version which would be msvcr120d.dll.

Comment by Alexander Gorrod [ 07/Apr/16 ]

kaloian.manassiev I've attempted to reproduce the failure you mention without success.

I've also reviewed the code, and it appears reasonable to me. It follows the pattern outlined on StackOverflow here: http://stackoverflow.com/questions/7369445/is-there-a-windows-equivalent-to-fdopen-for-handles, which appears to comply with the Windows documentation for the functionality.

Could you provide some additional information about the failure? I'm using an Evergreen Windows VS 2013 spawn host. I created a build as per your instructions in the ticket description. I started mongod from both a clean dbpath and a dbpath with an existing database (with and without content).

I notice that you use --dbg=on --opt=off in both the build steps, and the reason why Evergreen doesn't encounter the problem. Do I need to do something else to link against debug runtime libraries on Windows?

Generated at Thu Feb 08 04:03:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.