[SERVER-38680] WiredTiger error (13): Access is Denied after many read/write ops Created: 17/Dec/18  Updated: 08/Feb/23  Resolved: 04/Jan/19

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 4.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Chadwick Casper Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod.log    
Operating System: ALL
Steps To Reproduce:

2 Windows machines

One runs Windows 7 and a LabView C# driver for MongoDB

The other runs Windows 10 and hosts the database as a Network Service

Participants:

 Description   

Our database is hosted locally on a Windows machine. Connections through Compass are fine, and the database is able to receive write and read commands from another computer on the network through a LabView C# driver before crashing. After a few write/read iterations, WiredTiger throws this error:

 

2018-12-17T13:06:47.913-0800 E STORAGE [WTCheckpointThread] WiredTiger error (13) [1545080807:912513][8088:140729630932048], file:WiredTiger.wt, WT_SESSION.checkpoint: __win_fs_rename, 105: C:\Program Files\MongoDB\Server\4.0\data\WiredTiger.turtle.set to C:\Program Files\MongoDB\Server\4.0\data\WiredTiger.turtle: file-rename: MoveFileExW: Access is denied.
: Permission denied Raw: [1545080807:912513][8088:140729630932048], file:WiredTiger.wt, WT_SESSION.checkpoint: __win_fs_rename, 105: C:\Program Files\MongoDB\Server\4.0\data\WiredTiger.turtle.set to C:\Program Files\MongoDB\Server\4.0\data\WiredTiger.turtle: file-rename: MoveFileExW: Access is denied.
: Permission denied
2018-12-17T13:06:47.913-0800 E STORAGE [WTCheckpointThread] WiredTiger error (13) [1545080807:913052][8088:140729630932048], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_turtle_update, 397: WiredTiger.turtle: fatal turtle file update error: Permission denied Raw: [1545080807:913052][8088:140729630932048], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_turtle_update, 397: WiredTiger.turtle: fatal turtle file update error: Permission denied
2018-12-17T13:06:47.913-0800 E STORAGE [WTCheckpointThread] WiredTiger error (-31804) [1545080807:913052][8088:140729630932048], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1545080807:913052][8088:140729630932048], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-12-17T13:06:47.913-0800 F - [WTCheckpointThread] Fatal Assertion 50853 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 409
2018-12-17T13:06:47.913-0800 F - [WTCheckpointThread]
***aborting after fassert() failure
 
2018-12-17T13:06:47.928-0800 F - [WTJournalFlusher] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 65
2018-12-17T13:06:47.928-0800 F - [WTJournalFlusher]
***aborting after fassert() failure

 

Installed the drivers with the .msi file online, and the permissions seem to be set appropriately. Ie - Network Service is a permissions group that has full read and write control on the /data/db directory. The unofficial driver we're using to write from LabView is [here|https://github.com/RBXSystems/mongo-labview-driver.]

 



 Comments   
Comment by Daniel Hegener [ 04/Mar/19 ]

Just for completeness and tracking reasons: We are experiencing seeing the same logs on a random basis accompanied by a MongoDB crash all of which also appears to be related to some external component of which we have a lot: McAfee Scanner, McAfee DLP, IBM BigFix, FireEye, Bit9...

Comment by Danny Hatcher (Inactive) [ 04/Jan/19 ]

Hello Chad,

I wouldn't expect the firewall to be causing file issues so it seems that Webroot may have been the culprit. Regardless, I'm glad to hear that you solved your problem.

Danny

Comment by Chadwick Casper [ 04/Jan/19 ]

Hi Daniel,

 

Firewalls were disabled, but the error was still encountered. As a workaround, we're running Mongo on a virtual machine running Ubuntu. So far the system is reading and writing smoothly. It's not a solution to the original problem, but it seems to be a viable workaround for Windows systems with this file handling issue.

 

-Chad

Comment by Chadwick Casper [ 21/Dec/18 ]

Hi Daniel,

 

We have Webroot in addition to Windows Firewall both enabled on the PC. I'll try to disable one, the other, and both and let you know what comes of it.

 

-Chad

Comment by Danny Hatcher (Inactive) [ 21/Dec/18 ]

Hello Chad,

We've seen similar issues in the past because of virus scanners on the system. Do you have any processes other than mongod on the server that would be accessing those data files?

Thank you,

Danny

Comment by Chadwick Casper [ 18/Dec/18 ]

Hi Danny,

 

Thanks for handling the ticket. It seems like the server is able to restart itself after (usually after about a minute, as per Windows service settings). I attach a full log file for you to take a look at.

 

There's an additional CPython client in the full log file, which are accesses from a computer on the network running pymongo.

 

Best,

Chad

mongod.log

Comment by Danny Hatcher (Inactive) [ 18/Dec/18 ]

Hello Chadwick,

Thank you for your report. After the assertion mentioned in your snippet, are you able to start the server successfully or does it crash on every attempt? Please provide the full mongod log file covering the time period from before the first crash until now so that I can see if anything stands out.

Thank you,

Danny

Generated at Thu Feb 08 04:49:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.