[SERVER-32808] NTFS Compression causes "file system limitation" errors Created: 20/Jan/18  Updated: 20/May/23  Resolved: 02/Feb/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.6.2
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Gabriel Fair Assignee: Geert Bosch
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File ntfs_compression.png    
Sprint: Storage 2018-02-12
Participants:
Case:

 Description   

Importing to a new collection on a standalone database stored on a NTFS disk results in a WT library Panic. Restarting Mongodb after the panic also results in a panic. Both logs are below

Mongo: 3.6.2 Community
OS: Windows 10 Pro x64
Disk: NTFS with compression flag true
Processor: AMD FX-6300 6 Cores
Memory: 32gb DDR4
Database: WiredTiger
Sharding: No
Replication: No
Additional ServerConfig: maxIndexBuildMemoryUsageMegabytes: 30000
Pre import Collection Index: id (unique)
Import this jsonL Dataset: (59gb file) http://files.pushshift.io/reddit/comments/xz/RC_2015.xz

Import Error Log Section:

2018-01-19T21:15:48.543-0500 I COMMAND  [conn20] command reddit.fifteen_sixth_try command: insert { insert: "fifteen_sixth_try", ordered: false, $db: "reddit" } ninserted:1000 keysInserted:2000 numYields:0 reslen:29 locks:{ Global: { acquireCount: { r: 16, w: 16 } }, Database: { acquireCount: { w: 16 } }, Collection: { acquireCount: { w: 16 } } } protocol:op_msg 872ms
2018-01-19T21:15:49.278-0500 E STORAGE  [WTCheckpointThread] WiredTiger error (-31802) [1516414549:262111][13116:140726555009360], file:collection-0--4941774138393653127.wt, WT_SESSION.checkpoint: E:\gdrive\datalake\databases\mongodb\collection-0--4941774138393653127.wt: handle-write: WriteFile: failed to write 8192 bytes at offset 187563442176: The requested operation could not be completed due to a file system limitation
 
: WT_ERROR: non-specific WiredTiger error
2018-01-19T21:15:49.278-0500 E STORAGE  [WTCheckpointThread] WiredTiger error (22) [1516414549:278711][13116:140726555009360], file:index-1--4941774138393653127.wt, WT_SESSION.checkpoint: index-1--4941774138393653127.wt: the checkpoint failed, the system must restart: Invalid argument
2018-01-19T21:15:49.279-0500 E STORAGE  [WTCheckpointThread] WiredTiger error (-31804) [1516414549:278711][13116:140726555009360], file:index-1--4941774138393653127.wt, WT_SESSION.checkpoint: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-01-19T21:15:49.280-0500 F -        [WTCheckpointThread] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2018-01-19T21:15:49.280-0500 F -        [WTCheckpointThread] 
 
***aborting after fassert() failure
 
 
2018-01-19T21:15:49.524-0500 E STORAGE  [conn20] WiredTiger error (-31804) [1516414549:523854][13116:140726555009360], WT_SESSION.commit_transaction: journal/WiredTigerLog.0000013585: fatal log failure: WT_PANIC: WiredTiger library panic
2018-01-19T21:15:49.524-0500 E STORAGE  [thread46] WiredTiger error (-31804) [1516414549:523854][13116:140726555009360], log-server: log server error: WT_PANIC: WiredTiger library panic
2018-01-19T21:15:49.524-0500 F -        [thread46] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2018-01-19T21:15:49.525-0500 F -        [thread46] 
 
***aborting after fassert() failure
 
 
2018-01-19T21:15:49.524-0500 F -        [conn29] Fatal Assertion 28559 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 64
2018-01-19T21:15:49.525-0500 F -        [conn29] 
 
***aborting after fassert() failure
 
 
2018-01-19T21:15:49.524-0500 F -        [conn20] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2018-01-19T21:15:49.525-0500 F -        [conn20] 
 
***aborting after fassert() failure

And here are the restart logs:

2018-01-20T15:50:10.762-0500 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2018-01-20T15:50:10.762-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=15821M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-01-20T15:50:19.255-0500 I STORAGE  [initandlisten] WiredTiger message [1516481419:255018][3060:140726555009360], txn-recover: Main recovery loop: starting at 13579/91027712
2018-01-20T15:50:19.261-0500 I STORAGE  [initandlisten] WiredTiger message [1516481419:261018][3060:140726555009360], txn-recover: Recovering log 13579 through 13587
2018-01-20T15:50:20.472-0500 I STORAGE  [initandlisten] WiredTiger message [1516481420:472091][3060:140726555009360], file:index-2--4941774138393653127.wt, txn-recover: Recovering log 13580 through 13587
2018-01-20T15:50:25.112-0500 I STORAGE  [initandlisten] WiredTiger message [1516481425:111360][3060:140726555009360], file:sizeStorer.wt, txn-recover: Recovering log 13581 through 13587
2018-01-20T15:50:29.596-0500 I STORAGE  [initandlisten] WiredTiger message [1516481429:595623][3060:140726555009360], file:sizeStorer.wt, txn-recover: Recovering log 13582 through 13587
2018-01-20T15:50:34.473-0500 I STORAGE  [initandlisten] WiredTiger message [1516481434:472906][3060:140726555009360], file:sizeStorer.wt, txn-recover: Recovering log 13583 through 13587
2018-01-20T15:50:39.743-0500 I STORAGE  [initandlisten] WiredTiger message [1516481439:743213][3060:140726555009360], file:sizeStorer.wt, txn-recover: Recovering log 13584 through 13587
2018-01-20T15:50:44.477-0500 I STORAGE  [initandlisten] WiredTiger message [1516481444:476490][3060:140726555009360], file:sizeStorer.wt, txn-recover: Recovering log 13585 through 13587
2018-01-20T15:50:47.681-0500 I STORAGE  [initandlisten] WiredTiger message [1516481447:680677][3060:140726555009360], file:sizeStorer.wt, txn-recover: Recovering log 13586 through 13587
2018-01-20T15:50:48.275-0500 I STORAGE  [initandlisten] WiredTiger message [1516481448:275712][3060:140726555009360], file:sizeStorer.wt, txn-recover: Recovering log 13587 through 13587
2018-01-20T15:50:50.895-0500 E STORAGE  [thread1] WiredTiger error (-31802) [1516481450:894865][3060:140726555009360], file:collection-0--4941774138393653127.wt, eviction-server: E:\gdrive\datalake\databases\mongodb\collection-0--4941774138393653127.wt: handle-write: WriteFile: failed to write 16384 bytes at offset 187592658944: The requested operation could not be completed due to a file system limitation
 
: WT_ERROR: non-specific WiredTiger error
2018-01-20T15:50:50.895-0500 E STORAGE  [thread1] WiredTiger error (-31802) [1516481450:894865][3060:140726555009360], eviction-server: cache eviction thread error: WT_ERROR: non-specific WiredTiger error
2018-01-20T15:50:50.895-0500 E STORAGE  [thread1] WiredTiger error (-31804) [1516481450:894865][3060:140726555009360], eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-01-20T15:50:50.895-0500 F -        [thread1] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2018-01-20T15:50:50.895-0500 F -        [thread1] 
 
***aborting after fassert() failure
 
 
2018-01-20T15:50:50.897-0500 E STORAGE  [thread2] WiredTiger error (-31802) [1516481450:896866][3060:140726555009360], file:collection-0--4941774138393653127.wt, eviction-server: E:\gdrive\datalake\databases\mongodb\collection-0--4941774138393653127.wt: handle-write: WriteFile: failed to write 16384 bytes at offset 187592675328: The requested operation could not be completed due to a file system limitation
 
: WT_ERROR: non-specific WiredTiger error
2018-01-20T15:50:50.897-0500 E STORAGE  [thread2] WiredTiger error (-31802) [1516481450:897865][3060:140726555009360], eviction-server: cache eviction thread error: WT_ERROR: non-specific WiredTiger error
2018-01-20T15:50:50.901-0500 E STORAGE  [thread3] WiredTiger error (-31802) [1516481450:900865][3060:140726555009360], file:collection-0--4941774138393653127.wt, eviction-server: E:\gdrive\datalake\databases\mongodb\collection-0--4941774138393653127.wt: handle-write: WriteFile: failed to write 16384 bytes at offset 187592691712: The requested operation could not be completed due to a file system limitation
 
: WT_ERROR: non-specific WiredTiger error
2018-01-20T15:50:50.901-0500 E STORAGE  [thread3] WiredTiger error (-31802) [1516481450:900865][3060:140726555009360], eviction-server: cache eviction thread error: WT_ERROR: non-specific WiredTiger error
2018-01-20T15:50:50.901-0500 E STORAGE  [initandlisten] WiredTiger error (-31804) [1516481450:901865][3060:140726555009360], file:sizeStorer.wt, txn-recover: Recovery failed: WT_PANIC: WiredTiger library panic
2018-01-20T15:50:50.902-0500 F -        [initandlisten] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2018-01-20T15:50:50.902-0500 F -        [initandlisten] 
 
***aborting after fassert() failure



 Comments   
Comment by Geert Bosch [ 31/Jan/18 ]

Note that you can specify the --wiredTigerCollectionBlockCompressor=zlib option to enable the zlib compressor, which compresses a bit more slowly than the default compressor, but gives about 2x better compression in some local tests I did. This should get your final storage requirements to a similar level as you have now.

Comment by Gabriel Fair [ 31/Jan/18 ]

I see. Thank you. I'll make the change. An interesting note is how much space savings I'm getting from it. Screen cap below:

Comment by Geert Bosch [ 30/Jan/18 ]

From my research, it seems that this is related to the use of a compressed NTFS file system. In NTFS, there is a maximum number of fragments that a file may contain. So, larger files are more likely to run into this limit. Files using compression or deduplication will have more fragmentation due to blocks that are overwritten shrinking or expanding based on their contents. Finally, Windows cannot reduce fragmentation of files that are in use. So in this case, all risk factors for excessive fragmentation are present: very large files, compression and having a file that is kept open for extended periods.

The suggestion is to not use filesystem level compression for the partition that contains the MongoDB database. MongoDB's WiredTiger storage engine already provides compression (by default snappy, but can be configured to gzip for additional compression at the cost of speed), so additional filesystem level compression will not be very effective, or even counter-productive, as the filesystem compression take valuable memory and cpu resources away for little or no space savings.

See also: https://support.microsoft.com/en-us/help/967351/a-heavily-fragmented-file-in-an-ntfs-volume-may-not-grow-beyond-a-cert

Generated at Thu Feb 08 04:31:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.