[SERVER-18448] mongod start failed Created: 13/May/15  Updated: 14/May/15  Resolved: 14/May/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: naveen Assignee: Keith Bostic (Inactive)
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive SERVER-18448-fixed-2.zip     Zip Archive SERVER-18448-fixed.zip     HTML File WiredTiger     File WiredTiger.basecfg     File WiredTiger.lock     File WiredTiger.turtle     File WiredTiger.wt     File _mdb_catalog.wt     File sizeStorer.wt    
Issue Links:
Duplicate
duplicates SERVER-18316 Database with WT engine fails to reco... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

I am using mongodb 3.0.2, storage engine - wiredTiger
from yesterday morning i am facing this issue.
I went through all the links where hazard_max has to be set but it didn't work for me. please help me to get out of this issue.

2015-05-12T16:54:35.442+0530 I CONTROL ***** SERVER RESTARTED *****
2015-05-12T16:54:35.465+0530 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-05-12T16:54:35.469+0530 E STORAGE [initandlisten] WiredTiger (0) [1431429875:469521][10501:0x7f1a672e3c20], file:WiredTiger.wt, connection: read checksum error [4096B @ 32768, 3884066602 != 1841814112]
2015-05-12T16:54:35.469+0530 E STORAGE [initandlisten] WiredTiger (0) [1431429875:469550][10501:0x7f1a672e3c20], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-05-12T16:54:35.469+0530 E STORAGE [initandlisten] WiredTiger (-31804) [1431429875:469563][10501:0x7f1a672e3c20], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-05-12T16:54:35.469+0530 I - [initandlisten] Fatal Assertion 28558
2015-05-12T16:54:35.478+0530 I CONTROL [initandlisten] 
0xf68719 0xf06c91 0xeeaad1 0xd8761a 0x1399239 0x13993f5 0x1399894 0x12f0e52 0x130955c 0x1307358 0x13081d2 0x133014b 0x139877d 0x13669eb 0x132d937 0xd7150b 0xd6f3a8 0xa8033d 0x7f36a2 0x7f8c14 0x3bf4a1ed5d 0x7f1435
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B68719"},{"b":"400000","o":"B06C91"},{"b":"400000","o":"AEAAD1"},{"b":"400000","o":"98761A"},{"b":"400000","o":"F99239"},{"b":"400000","o":"F993F5"},{"b":"400000","o":"F99894"},{"b":"400000","o":"EF0E52"},{"b":"400000","o":"F0955C"},{"b":"400000","o":"F07358"},{"b":"400000","o":"F081D2"},{"b":"400000","o":"F3014B"},{"b":"400000","o":"F9877D"},{"b":"400000","o":"F669EB"},{"b":"400000","o":"F2D937"},{"b":"400000","o":"97150B"},{"b":"400000","o":"96F3A8"},{"b":"400000","o":"68033D"},{"b":"400000","o":"3F36A2"},{"b":"400000","o":"3F8C14"},{"b":"3BF4A00000","o":"1ED5D"},{"b":"400000","o":"3F1435"}],"processInfo":{ "mongodbVersion" : "3.0.2", "gitVersion" : "6201872043ecbbc0a4cc169b5482dcf385fc464f", "uname" : { "sysname" : "Linux", "release" : "2.6.32-504.12.2.el6.x86_64", "version" : "#1 SMP Wed Mar 11 22:03:14 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "528324F7E7422ECA899108CA2A12B5AAAA2A5569" }, { "b" : "7FFFB0EFE000", "elfType" : 3, "buildId" : "4B8E1260CCD7C3D8CC131E84001F7220651617EC" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B8DFF8E53D9F2B80C3C382E83EC17C828B536A39" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "934508308DAF0D5C61E9997463F0D8B0A3F096BA" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A4329A30669C783FA8DEEB7D1EA83749A8FA14E1" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "583411D8786F86A1D6B8741C502831E6122445A7" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "454F8FC6CC6502C6401E5F9E221564D80665D277" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "F07F2E7CF4BFB393CC9BBE8CDC6463652E14DB07" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "7D8E9374F4A4EA38A7C1E763F32240EA113E4208" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "246C3BAB0AB093AFD59D34C8CBF29E786DE4BE97" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "E4EAB3C200B7D8444FF95AB01F6466924A6A5F5F" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F8E59B70E469F3A924A268911FF8FD0C37E7460" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "54BA6B78A9220344E77463947215E42F0EABCC62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6797403AA5F8FAD8ADFF683478B45F528CE4FB0E" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "8CE28F280150E62296240E70ECAC64E4A57AB826" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "05733977F4E41652B86070B27A0CFC2C1EA7719D" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E3FA235F3BA3F776A01A18ECA737C9890F445923" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F8B68F301C19BF06AF56B4B06E0A69F89D2C1F8D" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf68719]
mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf06c91]
mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeeaad1]
mongod(+0x98761A) [0xd8761a]
mongod(__wt_eventv+0x489) [0x1399239]
mongod(__wt_err+0x95) [0x13993f5]
mongod(__wt_panic+0x24) [0x1399894]
mongod(__wt_bm_read+0x72) [0x12f0e52]
mongod(__wt_bt_read+0x1AC) [0x130955c]
mongod(__wt_btree_tree_open+0x58) [0x1307358]
mongod(__wt_btree_open+0xD02) [0x13081d2]
mongod(__wt_conn_btree_get+0x19B) [0x133014b]
mongod(__wt_session_get_btree+0x31D) [0x139877d]
mongod(__wt_metadata_open+0x2B) [0x13669eb]
mongod(wiredtiger_open+0xCD7) [0x132d937]
mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x2EB) [0xd7150b]
mongod(+0x96F3A8) [0xd6f3a8]
mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa8033d]
mongod(_ZN5mongo13initAndListenEi+0x422) [0x7f36a2]
mongod(main+0x134) [0x7f8c14]
libc.so.6(__libc_start_main+0xFD) [0x3bf4a1ed5d]
mongod(+0x3F1435) [0x7f1435]
----- END BACKTRACE -----
2015-05-12T16:54:35.478+0530 I - [initandlisten]
 
***aborting after fassert() failure

when i running command for increasing hazard then the problem remain
same.

[root@host name ~]# mongod --dbpath /home/mongodbProject/database/wiredTiger/ --storageEngine wiredTiger --wiredTigerEngineConfigString="hazard_max=20000"

2015-05-12T17:14:47.103+0530 I STORAGE Engine custom option: hazard_max=20000
2015-05-12T17:14:47.126+0530 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),hazard_max=20000
2015-05-12T17:14:47.130+0530 E STORAGE [initandlisten] WiredTiger (0) [1431431087:130057][14416:0x7f83a326bc20], file:WiredTiger.wt, connection: read checksum error [4096B @ 32768, 3884066602 != 1841814112]
2015-05-12T17:14:47.130+0530 E STORAGE [initandlisten] WiredTiger (0) [1431431087:130086][14416:0x7f83a326bc20], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-05-12T17:14:47.130+0530 E STORAGE [initandlisten] WiredTiger (-31804) [1431431087:130098][14416:0x7f83a326bc20], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-05-12T17:14:47.130+0530 I - [initandlisten] Fatal Assertion 28558
2015-05-12T17:14:47.139+0530 I CONTROL [initandlisten] 
0xf68719 0xf06c91 0xeeaad1 0xd8761a 0x1399239 0x13993f5 0x1399894 0x12f0e52 0x130955c 0x1307358 0x13081d2 0x133014b 0x139877d 0x13669eb 0x132d937 0xd7150b 0xd6f3a8 0xa8033d 0x7f36a2 0x7f8c14 0x3bf4a1ed5d 0x7f1435
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B68719"},{"b":"400000","o":"B06C91"},{"b":"400000","o":"AEAAD1"},{"b":"400000","o":"98761A"},{"b":"400000","o":"F99239"},{"b":"400000","o":"F993F5"},{"b":"400000","o":"F99894"},{"b":"400000","o":"EF0E52"},{"b":"400000","o":"F0955C"},{"b":"400000","o":"F07358"},{"b":"400000","o":"F081D2"},{"b":"400000","o":"F3014B"},{"b":"400000","o":"F9877D"},{"b":"400000","o":"F669EB"},{"b":"400000","o":"F2D937"},{"b":"400000","o":"97150B"},{"b":"400000","o":"96F3A8"},{"b":"400000","o":"68033D"},{"b":"400000","o":"3F36A2"},{"b":"400000","o":"3F8C14"},{"b":"3BF4A00000","o":"1ED5D"},{"b":"400000","o":"3F1435"}],"processInfo":{ "mongodbVersion" : "3.0.2", "gitVersion" : "6201872043ecbbc0a4cc169b5482dcf385fc464f", "uname" : { "sysname" : "Linux", "release" : "2.6.32-504.12.2.el6.x86_64", "version" : "#1 SMP Wed Mar 11 22:03:14 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "528324F7E7422ECA899108CA2A12B5AAAA2A5569" }, { "b" : "7FFFE011E000", "elfType" : 3, "buildId" : "4B8E1260CCD7C3D8CC131E84001F7220651617EC" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B8DFF8E53D9F2B80C3C382E83EC17C828B536A39" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "934508308DAF0D5C61E9997463F0D8B0A3F096BA" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A4329A30669C783FA8DEEB7D1EA83749A8FA14E1" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "583411D8786F86A1D6B8741C502831E6122445A7" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "454F8FC6CC6502C6401E5F9E221564D80665D277" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "F07F2E7CF4BFB393CC9BBE8CDC6463652E14DB07" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "7D8E9374F4A4EA38A7C1E763F32240EA113E4208" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "246C3BAB0AB093AFD59D34C8CBF29E786DE4BE97" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "E4EAB3C200B7D8444FF95AB01F6466924A6A5F5F" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F8E59B70E469F3A924A268911FF8FD0C37E7460" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "54BA6B78A9220344E77463947215E42F0EABCC62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6797403AA5F8FAD8ADFF683478B45F528CE4FB0E" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "8CE28F280150E62296240E70ECAC64E4A57AB826" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "05733977F4E41652B86070B27A0CFC2C1EA7719D" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E3FA235F3BA3F776A01A18ECA737C9890F445923" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F8B68F301C19BF06AF56B4B06E0A69F89D2C1F8D" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf68719]
mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf06c91]
mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeeaad1]
mongod(+0x98761A) [0xd8761a]
mongod(__wt_eventv+0x489) [0x1399239]
mongod(__wt_err+0x95) [0x13993f5]
mongod(__wt_panic+0x24) [0x1399894]
mongod(__wt_bm_read+0x72) [0x12f0e52]
mongod(__wt_bt_read+0x1AC) [0x130955c]
mongod(__wt_btree_tree_open+0x58) [0x1307358]
mongod(__wt_btree_open+0xD02) [0x13081d2]
mongod(__wt_conn_btree_get+0x19B) [0x133014b]
mongod(__wt_session_get_btree+0x31D) [0x139877d]
mongod(__wt_metadata_open+0x2B) [0x13669eb]
mongod(wiredtiger_open+0xCD7) [0x132d937]
mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x2EB) [0xd7150b]
mongod(+0x96F3A8) [0xd6f3a8]
mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa8033d]
mongod(_ZN5mongo13initAndListenEi+0x422) [0x7f36a2]
mongod(main+0x134) [0x7f8c14]
libc.so.6(__libc_start_main+0xFD) [0x3bf4a1ed5d]
mongod(+0x3F1435) [0x7f1435]
----- END BACKTRACE -----
2015-05-12T17:14:47.139+0530 I - [initandlisten]
 
***aborting after fassert() failure



 Comments   
Comment by Ramon Fernandez Marina [ 14/May/15 ]

jdiego@digital-legends.com, please open a new SERVER ticket with details about your specific case. It could be that you case is also a duplicate of other issues already being investigated, but we'll need to investigate it first.

Thanks,
Ramón.

Comment by Juan Manuel Diego G [ 14/May/15 ]

keith.bostic How can I fix the files my self, I have a non replicated mongod instance, shutted it down with "mongod --shutdown" command and when trying to restart it failed. I'm on 3.0.3 by the way and "mongod --repair --dbpath /var/lib/mongodb --storageEngine wiredTiger" failed.

Comment by Juan Manuel Diego G [ 14/May/15 ]

Ok, so I guess I'll have to remove all files of the dbPath to force full replica, do you know if the 3.0.3 has this fixed? because rackspace is in a reboot process right now, so it's possible some mongod processes will be terminated in a non elegant way.

Comment by Keith Bostic (Inactive) [ 14/May/15 ]

jdiego@digital-legends.com, the fixed files were unique to naveendahiya1992's case.

Comment by naveen [ 14/May/15 ]

Thanks for your suggestion, I will update mongodb as per stable version.

Comment by Juan Manuel Diego G [ 14/May/15 ]

keith.bostic, does this fix applies for anyone? or the fixed files are unique to naveen's case?

Comment by Keith Bostic (Inactive) [ 14/May/15 ]

naveendahiya1992, that's great to hear, thank you!

We believe you ran into a recovery bug in WiredTiger, triggered by your power failure.

The upcoming MongoDB releases will include fixes for this and a few related bugs, so we would strongly suggest upgrading to new stable releases of 3.0 as they come out.

Comment by naveen [ 14/May/15 ]

may I know the problem behind the issue and what type of modifications you had done to those file ?

Comment by naveen [ 14/May/15 ]

Thank you for your help. Data recovered successfully.

Comment by naveen [ 14/May/15 ]

before corruption everything was running fine, I have been using this database from last 4 weeks and on May 12th my system was restarted due to power fluctuations at that time mongod was running and then when i tried to restart mongod using "service mongod restart"
it gave me error of starting failed. Then I shared my error logs with you.

Comment by naveen [ 14/May/15 ]

Sorry, due to some problem I was unable to run it yesterday.
I run the backup command 2 hour ago, beacause my database is of nearly 60 gb , it is running from last two hours for every collection, and showing this log:

15-05-14T11:08:49.925+0530 I STORAGE [initandlisten] WiredTiger progress session.verify 1414280
2015-05-14T11:08:49.927+0530 I STORAGE [initandlisten] WiredTiger progress session.verify 1414290
2015-05-14T11:08:49.929+0530 I STORAGE [initandlisten] WiredTiger progress session.verify 1414300
2015-05-14T11:08:49.930+0530 I STORAGE [initandlisten] WiredTiger progress session.verify 1414310
2015-05-14T11:08:49.932+0530 I STORAGE [initandlisten] WiredTiger progress session.verify 1414320
.........
2015-05-14T11:08:49.934+0530 I STORAGE [initandlisten] WiredTiger progress session.verify 1414330
2015-05-14T11:08:50.173+0530 I STORAGE [initandlisten] WiredTiger progress session.verify 1414338
2015-05-14T11:08:50.173+0530 I STORAGE [initandlisten] Verify succeeded on uri table:collection-2--8482242995300467764. Not salvaging.
2015-05-14T11:08:50.707+0530 I INDEX [initandlisten] build index on: TraiData.fulldata properties: { v: 1, key:

{ _id: 1 }

, name: "id", ns: "TraiData.fulldata" }
2015-05-14T11:08:50.707+0530 I INDEX [initandlisten] building index using bulk method

I will let you know when it will complete.

Comment by Michael Cahill (Inactive) [ 14/May/15 ]

naveendahiya1992, is your system back up again? Do you need anything else from us?

Can you share any information about what happened before the files got into this state?

Comment by Keith Bostic (Inactive) [ 13/May/15 ]

naveendahiya1992, I've uploaded a second zip archive SERVER-18448-fixed-2 attachment that has repaired versions of both WiredTiger.wt and sizeStorer.wt.

If you put those files into place, and then run normal mongod --repair, that will hopefully take care of the problem.

Please let me know what happens, of course.

Comment by naveen [ 13/May/15 ]

unable to repair, getting similar type of error:

[root@host name wiredTiger]# mongod --dbpath /home/mongodbProject/database/wiredTiger --repair --repairpath /home/mongodbProject/repairData/rep --storageEngine wiredTiger
2015-05-13T16:30:15.206+0530 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7G,session_max=20000,eviction=(threads_max=4),statistics=(fast),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-05-13T16:30:15.546+0530 I STORAGE  [initandlisten] Repairing size cache
2015-05-13T16:30:15.554+0530 E STORAGE  [initandlisten] WiredTiger (0) [1431514815:554248][5946:0x7fcb2c444c20], file:sizeStorer.wt, session.verify: read checksum error [4096B @ 40960, 2019257843 != 0]
2015-05-13T16:30:15.554+0530 E STORAGE  [initandlisten] WiredTiger (0) [1431514815:554309][5946:0x7fcb2c444c20], file:sizeStorer.wt, session.verify: sizeStorer.wt: encountered an illegal file format or internal value
2015-05-13T16:30:15.554+0530 E STORAGE  [initandlisten] WiredTiger (-31804) [1431514815:554334][5946:0x7fcb2c444c20], file:sizeStorer.wt, session.verify: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-05-13T16:30:15.554+0530 I -        [initandlisten] Fatal Assertion 28558
2015-05-13T16:30:15.571+0530 I CONTROL  [initandlisten] 
 0xf68719 0xf06c91 0xeeaad1 0xd8761a 0x1399239 0x13993f5 0x1399894 0x12eeb3e 0x12eefd8 0x12f1bff 0x12f1d78 0x131b32a 0x139489e 0x1394ad8 0x1394f66 0xd6fc85 0xd71846 0xd6f3a8 0xa8033d 0x7f36a2 0x7f8c14 0x3bf4a1ed5d 0x7f1435
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B68719"},{"b":"400000","o":"B06C91"},{"b":"400000","o":"AEAAD1"},{"b":"400000","o":"98761A"},{"b":"400000","o":"F99239"},{"b":"400000","o":"F993F5"},{"b":"400000","o":"F99894"},{"b":"400000","o":"EEEB3E"},{"b":"400000","o":"EEEFD8"},{"b":"400000","o":"EF1BFF"},{"b":"400000","o":"EF1D78"},{"b":"400000","o":"F1B32A"},{"b":"400000","o":"F9489E"},{"b":"400000","o":"F94AD8"},{"b":"400000","o":"F94F66"},{"b":"400000","o":"96FC85"},{"b":"400000","o":"971846"},{"b":"400000","o":"96F3A8"},{"b":"400000","o":"68033D"},{"b":"400000","o":"3F36A2"},{"b":"400000","o":"3F8C14"},{"b":"3BF4A00000","o":"1ED5D"},{"b":"400000","o":"3F1435"}],"processInfo":{ "mongodbVersion" : "3.0.2", "gitVersion" : "6201872043ecbbc0a4cc169b5482dcf385fc464f", "uname" : { "sysname" : "Linux", "release" : "2.6.32-504.12.2.el6.x86_64", "version" : "#1 SMP Wed Mar 11 22:03:14 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "528324F7E7422ECA899108CA2A12B5AAAA2A5569" }, { "b" : "7FFF982F7000", "elfType" : 3, "buildId" : "4B8E1260CCD7C3D8CC131E84001F7220651617EC" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B8DFF8E53D9F2B80C3C382E83EC17C828B536A39" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "934508308DAF0D5C61E9997463F0D8B0A3F096BA" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A4329A30669C783FA8DEEB7D1EA83749A8FA14E1" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "583411D8786F86A1D6B8741C502831E6122445A7" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "454F8FC6CC6502C6401E5F9E221564D80665D277" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "F07F2E7CF4BFB393CC9BBE8CDC6463652E14DB07" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "7D8E9374F4A4EA38A7C1E763F32240EA113E4208" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "246C3BAB0AB093AFD59D34C8CBF29E786DE4BE97" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "E4EAB3C200B7D8444FF95AB01F6466924A6A5F5F" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F8E59B70E469F3A924A268911FF8FD0C37E7460" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "54BA6B78A9220344E77463947215E42F0EABCC62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6797403AA5F8FAD8ADFF683478B45F528CE4FB0E" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "8CE28F280150E62296240E70ECAC64E4A57AB826" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "05733977F4E41652B86070B27A0CFC2C1EA7719D" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E3FA235F3BA3F776A01A18ECA737C9890F445923" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F8B68F301C19BF06AF56B4B06E0A69F89D2C1F8D" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf68719]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf06c91]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeeaad1]
 mongod(+0x98761A) [0xd8761a]
 mongod(__wt_eventv+0x489) [0x1399239]
 mongod(__wt_err+0x95) [0x13993f5]
 mongod(__wt_panic+0x24) [0x1399894]
 mongod(__wt_block_extlist_read+0x6E) [0x12eeb3e]
 mongod(__wt_block_extlist_read_avail+0x28) [0x12eefd8]
 mongod(+0xEF1BFF) [0x12f1bff]
 mongod(__wt_block_verify_start+0x108) [0x12f1d78]
 mongod(__wt_verify+0x4AA) [0x131b32a]
 mongod(__wt_schema_worker+0x35E) [0x139489e]
 mongod(__wt_schema_worker+0x598) [0x1394ad8]
 mongod(+0xF94F66) [0x1394f66]
 mongod(_ZN5mongo18WiredTigerKVEngine16_salvageIfNeededEPKc+0x45) [0xd6fc85]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x626) [0xd71846]
 mongod(+0x96F3A8) [0xd6f3a8]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa8033d]
 mongod(_ZN5mongo13initAndListenEi+0x422) [0x7f36a2]
 mongod(main+0x134) [0x7f8c14]
 libc.so.6(__libc_start_main+0xFD) [0x3bf4a1ed5d]
 mongod(+0x3F1435) [0x7f1435]
-----  END BACKTRACE  -----
2015-05-13T16:30:15.571+0530 I -        [initandlisten] 
 
***aborting after fassert() failure

Comment by Michael Cahill (Inactive) [ 13/May/15 ]

I think this should be handled by normal mongod --repair .... Can you please follow the instructions at http://docs.mongodb.org/manual/tutorial/recover-data-following-unexpected-shutdown/ ?

Comment by naveen [ 13/May/15 ]

After changing permissions it executes and generate another error : file:sizeStorer.wt, session.open_cursor: read checksum error

error log:

2015-05-13T14:08:26.432+0530 I CONTROL  ***** SERVER RESTARTED *****
2015-05-13T14:08:26.463+0530 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-05-13T14:08:27.030+0530 E STORAGE  [initandlisten] WiredTiger (0) [1431506307:30312][4125:0x7f42a0284c20], file:sizeStorer.wt, session.open_cursor: read checksum error [4096B @ 32768, 4235238192 != 2481600301]
2015-05-13T14:08:27.030+0530 E STORAGE  [initandlisten] WiredTiger (0) [1431506307:30366][4125:0x7f42a0284c20], file:sizeStorer.wt, session.open_cursor: sizeStorer.wt: encountered an illegal file format or internal value
2015-05-13T14:08:27.030+0530 E STORAGE  [initandlisten] WiredTiger (-31804) [1431506307:30419][4125:0x7f42a0284c20], file:sizeStorer.wt, session.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-05-13T14:08:27.030+0530 I -        [initandlisten] Fatal Assertion 28558
2015-05-13T14:08:27.091+0530 I CONTROL  [initandlisten] 
 0xf68719 0xf06c91 0xeeaad1 0xd8761a 0x1399239 0x13993f5 0x1399894 0x12f0e52 0x130955c 0x1307358 0x13081d2 0x133014b 0x139877d 0x139887a 0x133d751 0x1396b5a 0x134edd3 0x13969a1 0x1396db5 0xd83424 0xd71740 0xd6f3a8 0xa8033d 0x7f36a2 0x7f8c14 0x3bf4a1ed5d 0x7f1435
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B68719"},{"b":"400000","o":"B06C91"},{"b":"400000","o":"AEAAD1"},{"b":"400000","o":"98761A"},{"b":"400000","o":"F99239"},{"b":"400000","o":"F993F5"},{"b":"400000","o":"F99894"},{"b":"400000","o":"EF0E52"},{"b":"400000","o":"F0955C"},{"b":"400000","o":"F07358"},{"b":"400000","o":"F081D2"},{"b":"400000","o":"F3014B"},{"b":"400000","o":"F9877D"},{"b":"400000","o":"F9887A"},{"b":"400000","o":"F3D751"},{"b":"400000","o":"F96B5A"},{"b":"400000","o":"F4EDD3"},{"b":"400000","o":"F969A1"},{"b":"400000","o":"F96DB5"},{"b":"400000","o":"983424"},{"b":"400000","o":"971740"},{"b":"400000","o":"96F3A8"},{"b":"400000","o":"68033D"},{"b":"400000","o":"3F36A2"},{"b":"400000","o":"3F8C14"},{"b":"3BF4A00000","o":"1ED5D"},{"b":"400000","o":"3F1435"}],"processInfo":{ "mongodbVersion" : "3.0.2", "gitVersion" : "6201872043ecbbc0a4cc169b5482dcf385fc464f", "uname" : { "sysname" : "Linux", "release" : "2.6.32-504.12.2.el6.x86_64", "version" : "#1 SMP Wed Mar 11 22:03:14 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "528324F7E7422ECA899108CA2A12B5AAAA2A5569" }, { "b" : "7FFFD3258000", "elfType" : 3, "buildId" : "4B8E1260CCD7C3D8CC131E84001F7220651617EC" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B8DFF8E53D9F2B80C3C382E83EC17C828B536A39" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "934508308DAF0D5C61E9997463F0D8B0A3F096BA" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A4329A30669C783FA8DEEB7D1EA83749A8FA14E1" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "583411D8786F86A1D6B8741C502831E6122445A7" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "454F8FC6CC6502C6401E5F9E221564D80665D277" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "F07F2E7CF4BFB393CC9BBE8CDC6463652E14DB07" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "7D8E9374F4A4EA38A7C1E763F32240EA113E4208" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "246C3BAB0AB093AFD59D34C8CBF29E786DE4BE97" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "E4EAB3C200B7D8444FF95AB01F6466924A6A5F5F" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F8E59B70E469F3A924A268911FF8FD0C37E7460" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "54BA6B78A9220344E77463947215E42F0EABCC62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6797403AA5F8FAD8ADFF683478B45F528CE4FB0E" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "8CE28F280150E62296240E70ECAC64E4A57AB826" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "05733977F4E41652B86070B27A0CFC2C1EA7719D" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E3FA235F3BA3F776A01A18ECA737C9890F445923" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F8B68F301C19BF06AF56B4B06E0A69F89D2C1F8D" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf68719]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf06c91]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeeaad1]
 mongod(+0x98761A) [0xd8761a]
 mongod(__wt_eventv+0x489) [0x1399239]
 mongod(__wt_err+0x95) [0x13993f5]
 mongod(__wt_panic+0x24) [0x1399894]
 mongod(__wt_bm_read+0x72) [0x12f0e52]
 mongod(__wt_bt_read+0x1AC) [0x130955c]
 mongod(__wt_btree_tree_open+0x58) [0x1307358]
 mongod(__wt_btree_open+0xD02) [0x13081d2]
 mongod(__wt_conn_btree_get+0x19B) [0x133014b]
 mongod(__wt_session_get_btree+0x31D) [0x139877d]
 mongod(__wt_session_get_btree_ckpt+0xBA) [0x139887a]
 mongod(__wt_curfile_open+0xE1) [0x133d751]
 mongod(__wt_open_cursor+0x26A) [0x1396b5a]
 mongod(__wt_curtable_open+0x2E3) [0x134edd3]
 mongod(__wt_open_cursor+0xB1) [0x13969a1]
 mongod(+0xF96DB5) [0x1396db5]
 mongod(_ZN5mongo20WiredTigerSizeStorerC1EP15__wt_connectionRKSs+0xB4) [0xd83424]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x520) [0xd71740]
 mongod(+0x96F3A8) [0xd6f3a8]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa8033d]
 mongod(_ZN5mongo13initAndListenEi+0x422) [0x7f36a2]
 mongod(main+0x134) [0x7f8c14]
 libc.so.6(__libc_start_main+0xFD) [0x3bf4a1ed5d]
 mongod(+0x3F1435) [0x7f1435]
-----  END BACKTRACE  -----
2015-05-13T14:08:27.091+0530 I -        [initandlisten] 
 
***aborting after fassert() failure

Comment by Michael Cahill (Inactive) [ 13/May/15 ]

That looks like a permissions issue: it might have been caused by decompressing and copying those files. Can you make sure that all the WiredTiger.* files in the database directory have the same owner, group and permissions? In other words, they look the same in ls -l apart from file size and date.

Comment by naveen [ 13/May/15 ]

After replacing files error occured : Operation not permitted

2015-05-13T13:54:07.121+0530 I CONTROL ***** SERVER RESTARTED *****
2015-05-13T13:54:07.148+0530 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-05-13T13:54:07.151+0530 E STORAGE [initandlisten] WiredTiger (1) [1431505447:151756][3971:0x7f88e52f3c20], file:WiredTiger.wt, connection: /home/mongodbProject/database/wiredTiger/WiredTiger.wt: Operation not permitted
2015-05-13T13:54:07.152+0530 I - [initandlisten] Assertion: 28595:1: Operation not permitted
2015-05-13T13:54:07.152+0530 I STORAGE [initandlisten] exception in initAndListen: 28595 1: Operation not permitted, terminating
2015-05-13T13:54:07.152+0530 I CONTROL [initandlisten] dbexit: rc: 100

Comment by Michael Cahill (Inactive) [ 13/May/15 ]

Can you try copying the contents of SERVER-18448-fixed.zip over the corresponding files in your database directory, then restarting mongod?

Please let me know whether this works, and I'm sorry for the inconvenience. Fixes for SERVER-18316 will be backported and released as soon as testing is complete.

Comment by naveen [ 13/May/15 ]

These are all files except collection.wt and index.wt
please find attachments

Comment by Michael Cahill (Inactive) [ 13/May/15 ]

Can you please upload all of the WiredTiger.* files from the database directory? Unfortunately, I can't give you a simple recipe to recover the WiredTiger.wt without a better understanding of what has gone wrong.

Comment by naveen [ 13/May/15 ]

this is only copy of data.
It looks like file:WiredTiger.wt is corrupted.
Is there any way to fix this issue or recover data ?

Comment by Michael Cahill (Inactive) [ 13/May/15 ]

This looks like SERVER-18316: did this start happening after the system was shut down while mongod was running?

What is your main goal: is this node part of a replica set or is the only copy of this data?

Generated at Thu Feb 08 03:47:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.