[SERVER-21732] v3.0.4+ not working on Windows in virtual environment missing AVX instruction set Created: 02/Dec/15  Updated: 19/Nov/16  Resolved: 02/Aug/16

Status: Closed
Project: Core Server
Component/s: Admin, Build
Affects Version/s: 3.0.4, 3.0.5, 3.0.6, 3.0.7, 3.2.0-rc5
Fix Version/s: 3.3.5

Type: Bug Priority: Major - P3
Reporter: Jan Scholtyssek Assignee: DO NOT USE - Backlog - Platform Team
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File AppCrash_mongod.exe_5267b2a0942511ae78e12bcdc5a071eedf1d948d_5db368ef_05040d6f.wer     Text File application_log_mongodb_v3.0.4+_crash_2015-12-02.txt     PNG File mongo_v3.0.4+_crash_screenshot_1.PNG     PNG File mongo_v3.0.4+_crash_screenshot_2.PNG     PNG File mongo_v3.0.4+_crash_systemInfo.PNG    
Issue Links:
Depends
depends on SERVER-19505 Add VS 2015 Builder Closed
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

1.)
Install version 3.0.4-rc0 or newer
on Windows
in virtual environment (preferably XenServer)

2.)
Double click mongod.exe, mongos.exe or mongo.exe (or start from command prompt)

Participants:

 Description   

For Windows on virtual machines: When installing a fresh copy of MongoDB starting with release version 3.0.4-rc0 all mongo-executables crash without message in command prompt but with Windows telling that the program stopped working.
Installation was done following the official documentation: https://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/

Environments tested:
Failing (mongod.exe and mongo.exe crash):
XenServer-virtual machines: Windows Server 2012 R2 / Windows 10 Pro (both 64bit)
Success (mongod.exe and mongo.exe start as expected):
Real PCs: Windows 8.1 Pro / Windows 10 Enterprise (both 64-bit)

MongoDB v3.0.3 starts as expected on all systems. v2.6.11 starts as expected on all systems. ("Affected Verions" lists the tested versions).
Tested with both firewall on and off
Tested with administrator privilleges.
Attachment shows Windows crash report and system information for Windows Server 2012 R2 on XenServer VM.



 Comments   
Comment by Jan Scholtyssek [ 02/Aug/16 ]

We have tested with 3.3.10 on XenServer and can confirm that the issue is fixed. Thank you so much for your support and development.

Comment by Mark Benvenuto [ 02/Aug/16 ]

With the upgrade in compilers to VS 2015 Update 2, this has been fixed according to Microsoft Connect.

Comment by Mark Benvenuto [ 03/Dec/15 ]

One correction, it is possible to compile MongoDB 3.0.x and later without tcmalloc by specifying "--allocator=system".

Comment by Mark Benvenuto [ 03/Dec/15 ]

Jan, I am afraid that only Visual Studio 2013 is supported at this time for compiling MongoDB. The codes does not compile with Visual Studio 2015 (also called version 14 in the connect issue) at this time. See SERVER-19505.

Comment by Jan Scholtyssek [ 03/Dec/15 ]

Hi Mark.
Thank you very much for the fast investigation and response.
I will test if using a newer compiler can solve the issue, report back (can take some days or weeks) and stick with v3.0.3 on the XenServer machines until then.

Comment by Mark Benvenuto [ 03/Dec/15 ]

Thank for the information. The cause of the issue is that mathematical function log() in Microsoft Visual C++ 2013 (the compiler we compile the code with) depends on vpsrlq which is a part of the AVX instruction set. It appears either that your processor or Xen Server does not support it. In 3.0.4-rc0, we switched to using TCMalloc on Windows for WiredTiger which calls log() as part of startup.

The instruction in question which triggered c000001d, i.e., Illegal Instruction:

0:000> u 00007ff8`46170000+0000000000092bc3
MSVCR120!log+0x2d3 [f:\dd\vctools\crt\fpw32\tran\amd64\log.asm @ 429]:
00007ff8`46202bc3 c5e173d034      vpsrlq  xmm3,xmm0,34h

It is fixed in a later version of the compiler which we have not adopted yet at this time.

https://connect.microsoft.com/VisualStudio/feedback/details/987093/x64-log-function-uses-vpsrlq-avx-instruction-without-regard-to-operating-system-so-it-crashes-on-vista-x64

Comment by Jan Scholtyssek [ 03/Dec/15 ]

WER Report is attached.

Comment by Mark Benvenuto [ 02/Dec/15 ]

pantojasc As part of the crash report by WER, there should be a set of files logged in a path when it crashes. The path varies for each crash, but you can find the information in the dialog box/text report. If you could please attach a set of these files from a particular crash, it would help us investigate.

Example:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_mongod.exe_5267b2a0942511ae78e12bcdc5a071eedf1d948d_5db368ef_05040d6f

Comment by Jan Scholtyssek [ 02/Dec/15 ]

Same issue when using unpacked zip-archive (v3.0.7 without SSL). No output in console. Immediate "stopped working" message, Windows "Application" log same as attached.

Comment by Jan Scholtyssek [ 02/Dec/15 ]

Package used is MSI for all versions tested. Except for 3.0.7 all without SSL-extension.
No configuration used in testing. The command prompt was always empty. In a single test a configuration file was used with v3.0.7. The log-file was empty (file created manually beforehand).
Will test with zip-file and report on result.

Comment by Ramon Fernandez Marina [ 02/Dec/15 ]

pantojasc, what we need to see is the logs produced by mongod.exe. If launched from the command line as mongod.exe the logs will be displayed directly; if you're using a configuration file check to see where they are being sent.

Can you also please comment on which package did you use to install MongDB, the MSI or the ZIP one?

Thanks,
Ramón.

Comment by Jan Scholtyssek [ 02/Dec/15 ]

Attached is the Windows Event Viewer "Application" log, showing messages from the install and first launch of mongod.exe.
OS: Windows Server 2012 R2
Environment: XenServer VM

Comment by Ramon Fernandez Marina [ 02/Dec/15 ]

Did you use the MSI package or the zip files? Can you please upload the server logs for the failed installations? They should contain some indication as to what the issue is.

Thanks,
Ramón.

Comment by Jan Scholtyssek [ 02/Dec/15 ]

Tested with Windows 8.1 Pro in Oracle Virtualbox: No issues. (version 3.0.7 and 3.2.0-rc5).
Issue might be restricted to XenServer VMs.

Comment by Jan Scholtyssek [ 02/Dec/15 ]

Tested versions without the issue:
2.6.11, 3.0.0. 3.0.3

Generated at Thu Feb 08 03:58:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.