Loading...

Type: Bug
Resolution: Incomplete
Priority: Major - P3
Fix Version/s: None
Affects Version/s: 3.0.7
Component/s: WiredTiger
Labels:
None

Operating System:
Linux
Confidence Status:
None
Work Order:
3

Aha! Reference:
None
Tracking Level:
None
Risk Status:
None
Exec Notes:
None
Goal Name:
None
Goal Link:
None

I am using mongodb 3.0.7, storage engine - wiredTiger
My server unexpected power down and the mongod instance stops unexpectedly,now I start cann‘t start mongodb and repair always with error

[2016-01-29 08:51:47] mongodb mongodb.conf is dead,restarting..
2016-01-29T08:51:47.077+0800 I CONTROL  ***** SERVER RESTARTED *****
2016-01-29T08:51:47.116+0800 W -        [initandlisten] Detected unclean shutdown - /home/zenic/mongodb/data/mongod.lock is not empty.
2016-01-29T08:51:47.116+0800 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2016-01-29T08:51:47.116+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-01-29T08:51:47.132+0800 E STORAGE  [initandlisten] WiredTiger (0) [1454028707:132450][17057:0x7f130a908c80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 102400: block header checksum of 1600484457 doesn't match expected checksum of 12792921
2016-01-29T08:51:47.132+0800 E STORAGE  [initandlisten] WiredTiger (0) [1454028707:132759][17057:0x7f130a908c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2016-01-29T08:51:47.132+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1454028707:132820][17057:0x7f130a908c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-01-29T08:51:47.132+0800 I -        [initandlisten] Fatal Assertion 28558
2016-01-29T08:51:47.147+0800 I CONTROL  [initandlisten] 
 0xf758c9 0xf12561 0xef63e1 0xd94a6a 0x13aab79 0x13aad35 0x13ab1d4 0x12fc0ae 0x12fc603 0x12f93c8 0x12fd328 0x1316da2 0x134070b 0x13a9d28 0x1377adb 0x133ddc3 0xd7ea3f 0xd7c5b8 0xa7e8ed 0x7f5806 0x7fabc9 0x7f1308ec5af5 0x7f378d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B758C9"},{"b":"400000","o":"B12561"},{"b":"400000","o":"AF63E1"},{"b":"400000","o":"994A6A"},{"b":"400000","o":"FAAB79"},{"b":"400000","o":"FAAD35"},{"b":"400000","o":"FAB1D4"},{"b":"400000","o":"EFC0AE"},{"b":"400000","o":"EFC603"},{"b":"400000","o":"EF93C8"},{"b":"400000","o":"EFD328"},{"b":"400000","o":"F16DA2"},{"b":"400000","o":"F4070B"},{"b":"400000","o":"FA9D28"},{"b":"400000","o":"F77ADB"},{"b":"400000","o":"F3DDC3"},{"b":"400000","o":"97EA3F"},{"b":"400000","o":"97C5B8"},{"b":"400000","o":"67E8ED"},{"b":"400000","o":"3F5806"},{"b":"400000","o":"3FABC9"},{"b":"7F1308EA4000","o":"21AF5"},{"b":"400000","o":"3F378D"}],"processInfo":{ "mongodbVersion" : "3.0.7", "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd", "uname" : { "sysname" : "Linux", "release" : "3.10.0-123.el7.x86_64", "version" : "#1 SMP Mon Jun 30 12:09:22 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "E359E52F4A5E2A16804BCA34965CC29081397CDF" }, { "b" : "7FFF912FE000", "elfType" : 3, "buildId" : "D7952DC468957C2B14B6BB79E613D48BA1224706" }, { "b" : "7F130A4E1000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "18562EE0363BC9BD7101610BD86469AA426D0C44" }, { "b" : "7F130A275000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "58FEDFFED1A388AD9E495F9A6C91A851B9537765" }, { "b" : "7F1309E90000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "F214E8640FDA5097E7A90CE7974B3FF76C6C42D9" }, { "b" : "7F1309C88000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "8832E3070AB0758762836EEC8FCDDEDEF8235340" }, { "b" : "7F1309A84000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "B7C4BC0854BF5DE16B535353B38235CA42349C1E" }, { "b" : "7F130977D000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "63C62D6263FF98E6DD6896CB3E716E499744A4C9" }, { "b" : "7F130947B000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "D70EAB176DDA46DE292FEB8208A0E8A6718BAF3B" }, { "b" : "7F1309265000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "07120A9AC1BF3BCDD4A3EA1E0C47234A4A5C84F9" }, { "b" : "7F1308EA4000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "78186287BBA77069A056A5CCBEB14B7FD2CA3A4B" }, { "b" : "7F130A6FD000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "4EADCA6CB82E0A85EDB87C15B5E3980742514501" }, { "b" : "7F1308C5A000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "641A441AB91715A7E3AF8AD9AF38EE07F17866FE" }, { "b" : "7F130897A000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "08E8BA638E79EC07F98198ED40F90FA87D5EEEB5" }, { "b" : "7F1308776000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D2678F5F391BF2877E1BD6FAD16DBC589ED0BBF3" }, { "b" : "7F1308541000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "8269E77C68B707158D2B1BEA356EE0FC2A1C0024" }, { "b" : "7F130832B000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "E45643F27F3B3E960F3691AFC6EC27A98EF7B46B" }, { "b" : "7F130811D000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "577A21CDAA3D662B87D53AFAA12A1E7B34AD513F" }, { "b" : "7F1307F19000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F1307CFF000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "519F19CF966514EAC9B25BE6FE953750E466D3C1" }, { "b" : "7F1307ADA000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, { "b" : "7F1307879000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "B19961A753FDFF85BD071340139A7F024BAEFFCA" }, { "b" : "7F1307654000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "218D03D1F6CF1A099A4D467B5E8ECF4F2BF45750" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf758c9]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf12561]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xef63e1]
 mongod(+0x994A6A) [0xd94a6a]
 mongod(__wt_eventv+0x489) [0x13aab79]
 mongod(__wt_err+0x95) [0x13aad35]
 mongod(__wt_panic+0x24) [0x13ab1d4]
 mongod(__wt_block_extlist_read+0x6E) [0x12fc0ae]
 mongod(__wt_block_extlist_read_avail+0x33) [0x12fc603]
 mongod(__wt_block_checkpoint_load+0x1C8) [0x12f93c8]
 mongod(+0xEFD328) [0x12fd328]
 mongod(__wt_btree_open+0xAA2) [0x1316da2]
 mongod(__wt_conn_btree_get+0x19B) [0x134070b]
 mongod(__wt_session_get_btree+0x3F8) [0x13a9d28]
 mongod(__wt_metadata_open+0x2B) [0x1377adb]
 mongod(wiredtiger_open+0xCD3) [0x133ddc3]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x52F) [0xd7ea3f]
 mongod(+0x97C5B8) [0xd7c5b8]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa7e8ed]
 mongod(_ZN5mongo13initAndListenEi+0x246) [0x7f5806]
 mongod(main+0x139) [0x7fabc9]
 libc.so.6(__libc_start_main+0xF5) [0x7f1308ec5af5]
 mongod(+0x3F378D) [0x7f378d]
-----  END BACKTRACE  -----
2016-01-29T08:51:47.147+0800 I -        [initandlisten] 

***aborting after fassert() failure


14
[2016-01-29 08:51:47] mongodb mongodb.conf is running..

when delete the mongod.lock file and running command for repair then the problem remain

[root@sdn32 bin]# ./mongod --dbpath /home/zenic/mongodb/data --storageEngine wiredTiger --repair --repairpath /home/zenic/mongodb/data0

2016-01-29T08:59:31.518+0800 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2016-01-29T08:59:31.518+0800 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-01-29T08:59:31.533+0800 E STORAGE  [initandlisten] WiredTiger (0) [1454029171:533420][13575:0x7fd440366c80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 102400: block header checksum of 1600484457 doesn't match expected checksum of 12792921
2016-01-29T08:59:31.533+0800 E STORAGE  [initandlisten] WiredTiger (0) [1454029171:533611][13575:0x7fd440366c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2016-01-29T08:59:31.533+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1454029171:533683][13575:0x7fd440366c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-01-29T08:59:31.533+0800 I -        [initandlisten] Fatal Assertion 28558
2016-01-29T08:59:31.549+0800 I CONTROL  [initandlisten] 
 0xf758c9 0xf12561 0xef63e1 0xd94a6a 0x13aab79 0x13aad35 0x13ab1d4 0x12fc0ae 0x12fc603 0x12f93c8 0x12fd328 0x1316da2 0x134070b 0x13a9d28 0x1377adb 0x133ddc3 0xd7e90c 0xd7c5b8 0xa7e8ed 0x7f5806 0x7fabc9 0x7fd43e923af5 0x7f378d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B758C9"},{"b":"400000","o":"B12561"},{"b":"400000","o":"AF63E1"},{"b":"400000","o":"994A6A"},{"b":"400000","o":"FAAB79"},{"b":"400000","o":"FAAD35"},{"b":"400000","o":"FAB1D4"},{"b":"400000","o":"EFC0AE"},{"b":"400000","o":"EFC603"},{"b":"400000","o":"EF93C8"},{"b":"400000","o":"EFD328"},{"b":"400000","o":"F16DA2"},{"b":"400000","o":"F4070B"},{"b":"400000","o":"FA9D28"},{"b":"400000","o":"F77ADB"},{"b":"400000","o":"F3DDC3"},{"b":"400000","o":"97E90C"},{"b":"400000","o":"97C5B8"},{"b":"400000","o":"67E8ED"},{"b":"400000","o":"3F5806"},{"b":"400000","o":"3FABC9"},{"b":"7FD43E902000","o":"21AF5"},{"b":"400000","o":"3F378D"}],"processInfo":{ "mongodbVersion" : "3.0.7", "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd", "uname" : { "sysname" : "Linux", "release" : "3.10.0-123.el7.x86_64", "version" : "#1 SMP Mon Jun 30 12:09:22 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "E359E52F4A5E2A16804BCA34965CC29081397CDF" }, { "b" : "7FFF76229000", "elfType" : 3, "buildId" : "D7952DC468957C2B14B6BB79E613D48BA1224706" }, { "b" : "7FD43FF3F000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "18562EE0363BC9BD7101610BD86469AA426D0C44" }, { "b" : "7FD43FCD3000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "58FEDFFED1A388AD9E495F9A6C91A851B9537765" }, { "b" : "7FD43F8EE000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "F214E8640FDA5097E7A90CE7974B3FF76C6C42D9" }, { "b" : "7FD43F6E6000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "8832E3070AB0758762836EEC8FCDDEDEF8235340" }, { "b" : "7FD43F4E2000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "B7C4BC0854BF5DE16B535353B38235CA42349C1E" }, { "b" : "7FD43F1DB000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "63C62D6263FF98E6DD6896CB3E716E499744A4C9" }, { "b" : "7FD43EED9000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "D70EAB176DDA46DE292FEB8208A0E8A6718BAF3B" }, { "b" : "7FD43ECC3000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "07120A9AC1BF3BCDD4A3EA1E0C47234A4A5C84F9" }, { "b" : "7FD43E902000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "78186287BBA77069A056A5CCBEB14B7FD2CA3A4B" }, { "b" : "7FD44015B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "4EADCA6CB82E0A85EDB87C15B5E3980742514501" }, { "b" : "7FD43E6B8000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "641A441AB91715A7E3AF8AD9AF38EE07F17866FE" }, { "b" : "7FD43E3D8000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "08E8BA638E79EC07F98198ED40F90FA87D5EEEB5" }, { "b" : "7FD43E1D4000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D2678F5F391BF2877E1BD6FAD16DBC589ED0BBF3" }, { "b" : "7FD43DF9F000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "8269E77C68B707158D2B1BEA356EE0FC2A1C0024" }, { "b" : "7FD43DD89000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "E45643F27F3B3E960F3691AFC6EC27A98EF7B46B" }, { "b" : "7FD43DB7B000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "577A21CDAA3D662B87D53AFAA12A1E7B34AD513F" }, { "b" : "7FD43D977000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7FD43D75D000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "519F19CF966514EAC9B25BE6FE953750E466D3C1" }, { "b" : "7FD43D538000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, { "b" : "7FD43D2D7000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "B19961A753FDFF85BD071340139A7F024BAEFFCA" }, { "b" : "7FD43D0B2000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "218D03D1F6CF1A099A4D467B5E8ECF4F2BF45750" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf758c9]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf12561]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xef63e1]
 mongod(+0x994A6A) [0xd94a6a]
 mongod(__wt_eventv+0x489) [0x13aab79]
 mongod(__wt_err+0x95) [0x13aad35]
 mongod(__wt_panic+0x24) [0x13ab1d4]
 mongod(__wt_block_extlist_read+0x6E) [0x12fc0ae]
 mongod(__wt_block_extlist_read_avail+0x33) [0x12fc603]
 mongod(__wt_block_checkpoint_load+0x1C8) [0x12f93c8]
 mongod(+0xEFD328) [0x12fd328]
 mongod(__wt_btree_open+0xAA2) [0x1316da2]
 mongod(__wt_conn_btree_get+0x19B) [0x134070b]
 mongod(__wt_session_get_btree+0x3F8) [0x13a9d28]
 mongod(__wt_metadata_open+0x2B) [0x1377adb]
 mongod(wiredtiger_open+0xCD3) [0x133ddc3]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x3FC) [0xd7e90c]
 mongod(+0x97C5B8) [0xd7c5b8]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa7e8ed]
 mongod(_ZN5mongo13initAndListenEi+0x246) [0x7f5806]
 mongod(main+0x139) [0x7fabc9]
 libc.so.6(__libc_start_main+0xF5) [0x7fd43e923af5]
 mongod(+0x3F378D) [0x7f378d]
-----  END BACKTRACE  -----
2016-01-29T08:59:31.549+0800 I -        [initandlisten] 

***aborting after fassert() failure

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

_mdb_catalog.wt
36 kB
Jan 29 2016 03:01:37 AM UTC
mongodb.conf
0.2 kB
Jan 29 2016 03:01:37 AM UTC
repair_attempt.tgz
7 kB
Feb 05 2016 02:14:19 PM UTC
sizeStorer.wt
36 kB
Jan 29 2016 03:01:37 AM UTC
WiredTiger.lock
0.0 kB
Jan 29 2016 03:01:37 AM UTC
WiredTiger.turtle
0.9 kB
Jan 29 2016 03:01:37 AM UTC
WiredTiger.wt
152 kB
Jan 29 2016 03:01:37 AM UTC

Details

Description

Attachments

Attachments

Activity

People

Dates