[SERVER-27948] Mongodb crashed with "File too large" Created: 08/Feb/17  Updated: 27/May/19  Resolved: 09/Feb/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alps Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Operating System: ALL
Participants:
Case:

 Description   

Hi There

we have a 8TB db. and Yesterday db crashed without any reasons. after dig into log and we are seeing the below errors.

i tried to restart that node,it won't start.also we had replication on another server but that server e crashed too. please take a look

2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (27) [1486569324:661608][9029:0x7f8bef870700], file:collection-9-2207001049530720609.wt, eviction-worker: collection-9-2207001049530720609.wt: handle-write: pwrite: failed to write 4096 bytes at offset 2194719883264: File too large
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (27) [1486569324:661933][9029:0x7f8bef870700], eviction-worker: cache eviction worker error: File too large
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (-31804) [1486569324:661979][9029:0x7f8bef870700], eviction-worker: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-02-08T09:55:24.662-0600 I - [thread1] Fatal Assertion 28558
2017-02-08T09:55:24.662-0600 E STORAGE [initandlisten] WiredTiger (-31804) [1486569324:661994][9029:0x7f8bf3298cc0], file:sizeStorer.wt, txn-recover: Operation failed during recovery: WT_PANIC: WiredTiger library panic
2017-02-08T09:55:24.662-0600 I - [thread1]



 Comments   
Comment by Alps [ 09/Feb/17 ]

Thanks Thomas. I am moving our cluster to a XFS filesystem. will get back to you later.

Thanks

Comment by Kelsey Schubert [ 09/Feb/17 ]

Hi alps,

Thanks for the additional information. I would recommend changing your filesystem according to our production notes. In addition, you may want to confirm to that the ulimit settings are configured correctly.

From your responses, I do not see anything to indicate a bug in the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

Kind regards,
Thomas

Comment by Alps [ 09/Feb/17 ]
  1. file -sL /dev/sda4
    /dev/sda4: Linux rev 1.0 *ext3 *filesystem data, UUID=24b81ccc-a0d3-42dd-b3f7-d1284d21df1f, volume name "root" (needs journal recovery) (large files)

tune2fs 1.42.9 (4-Feb-2014)
Filesystem volume name: root
Last mounted on: /
Filesystem UUID: 24b81ccc-a0d3-42dd-b3f7-d1284d21df1f
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 244084736
Block count: 1952671488
Reserved block count: 97633574
Free blocks: 1936675470
Free inodes: 243952988
First block: 0
*Block size: 4096
*Fragment size: 4096

Comment by Alps [ 09/Feb/17 ]

We may found the issue. we are using EXT3 and Block Size is 4096 Bytes. according to wikipedia, maximum file size is 2TiB if block size is 4TiB, is that true ? would this be the root problem ? please advise
https://en.wikipedia.org/wiki/Ext3

Thanks

Comment by Alps [ 09/Feb/17 ]

any words ?

Thanks

Comment by Alps [ 08/Feb/17 ]

just fyi, collection-9-2207001049530720609.wt is 2.0TB

Thanks

Comment by Alps [ 08/Feb/17 ]

2017-02-08T10:58:35.004-0600 I -        [initandlisten] Detected data files in /opt/R1/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-02-08T10:58:35.004-0600 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=15G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-02-08T10:58:38.297-0600 E STORAGE  [thread1] WiredTiger (27) [1486573118:296987][9532:0x7f4f6a07e700], file:collection-9-2207001049530720609.wt, eviction-worker: collection-9-2207001049530720609.wt: handle-write: pwrite: failed to write 4096 bytes at offset 2194719883264: File too large
2017-02-08T10:58:38.297-0600 E STORAGE  [thread1] WiredTiger (27) [1486573118:297392][9532:0x7f4f6a07e700], eviction-worker: cache eviction worker error: File too large
2017-02-08T10:58:38.297-0600 E STORAGE  [thread1] WiredTiger (-31804) [1486573118:297433][9532:0x7f4f6a07e700], eviction-worker: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-02-08T10:58:38.297-0600 I -        [thread1] Fatal Assertion 28558
2017-02-08T10:58:38.297-0600 I -        [thread1]
 
***aborting after fassert() failure
 
 
2017-02-08T10:58:38.297-0600 E STORAGE  [initandlisten] WiredTiger (-31804) [1486573118:297567][9532:0x7f4f6caa4cc0], file:sizeStorer.wt, txn-recover: Operation failed during recovery: WT_PANIC: WiredTiger library panic
2017-02-08T10:58:38.297-0600 I -        [initandlisten] Fatal Assertion 28558
2017-02-08T10:58:38.297-0600 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2017-02-08T10:58:38.304-0600 F -        [thread1] Got signal: 6 (Aborted).
 
 0x1315982 0x1314ad9 0x13152e2 0x7f4f6b71f340 0x7f4f6b380cc9 0x7f4f6b3840d8 0x129f992 0x109ad23 0x1a7676c 0x1a76c2d 0x1a77014 0x1a22473 0x7f4f6b717182 0x7f4f6b44447d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F15982","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F14AD9"},{"b":"400000","o":"F152E2"},{"b":"7F4F6B70F000","o":"10340"},{"b":"7F4F6B34A000","o":"36CC9","s":"gsignal"},{"b":"7F4F6B34A000","o":"3A0D8","s":"abort"},{"b":"400000","o":"E9F992","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C9AD23"},{"b":"400000","o":"167676C","s":"__wt_eventv"},{"b":"400000","o":"1676C2D","s":"__wt_err"},{"b":"400000","o":"1677014","s":"__wt_panic"},{"b":"400000","o":"1622473"},{"b":"7F4F6B70F000","o":"8182"},{"b":"7F4F6B34A000","o":"FA47D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.6", "gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.2.0-36-generic", "version" : "#41~14.04.1-Ubuntu SMP Tue Apr 19 17:03:32 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B35B0D2B9CB934315BA086AB5C88C919B9B5BA50" }, { "b" : "7FFFFD9CD000", "elfType" : 3, "buildId" : "8B2559B12BD6E6C7212A35F980FB4E639F6DAB3B" }, { "b" : "7F4F6C631000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7F4F6C255000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF8351B730830BDBCE0DCABBE06574B7144" }, { "b" : "7F4F6C04D000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "B376100CAB1EAC4E5DE066EACFC282BF7C0B54F3" }, { "b" : "7F4F6BE49000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "67699FFDA9FD2A552032E0652A242E82D65AA10D" }, { "b" : "7F4F6BB43000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "EF3F6DFFA1FBE48436EC6F45CD3AABA157064BB4" }, { "b" : "7F4F6B92D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F4F6B70F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "AF06068681750736E0524DF17D5A86CB2C3F765C" }, { "b" : "7F4F6B34A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "5382058B69031CAA9B9996C11061CD164C9398FF" }, { "b" : "7F4F6C890000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2A816C3EBBA4E12813FBD34B06FBD25BC892A67F" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1315982]
 mongod(+0xF14AD9) [0x1314ad9]
 mongod(+0xF152E2) [0x13152e2]
 libpthread.so.0(+0x10340) [0x7f4f6b71f340]
 libc.so.6(gsignal+0x39) [0x7f4f6b380cc9]
 libc.so.6(abort+0x148) [0x7f4f6b3840d8]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x129f992]
 mongod(+0xC9AD23) [0x109ad23]
 mongod(__wt_eventv+0x42C) [0x1a7676c]
 mongod(__wt_err+0x8D) [0x1a76c2d]
 mongod(__wt_panic+0x24) [0x1a77014]
 mongod(+0x1622473) [0x1a22473]
 libpthread.so.0(+0x8182) [0x7f4f6b717182]
 libc.so.6(clone+0x6D) [0x7f4f6b44447d]
-----  END BACKTRACE  -----

Generated at Thu Feb 08 04:16:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.