Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16596

Repairing a WiredTiger database with a flipped bit triggers a checksum fassert

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Storage
    • Fully Compatible
    • ALL
    • Hide

      1) Create the uncorrupted database(attached as uncorrupted.tar.bz2):

      ./mongod --directoryperdb --dbpath corruption --storageEngine wiredtiger
> use db
switched to db db
> db.collection.insert({data: true})

      2) Set up database db for mdb-wt:

      sajack@linux-p7yj /home/sajack/mongo git master () % cd corruption/db
sajack@linux-p7yj /home/sajack/mongo/corruption/db git master () % ls
collection-0--6386865262002950881.wt  index-1--6386865262002950881.wt
sajack@linux-p7yj /home/sajack/mongo/corruption/db git master () % ln -s ../WiredTiger.wt .
sajack@linux-p7yj /home/sajack/mongo/corruption/db git master () % ln -s ../WiredTiger.turtle .

      3) Find offsets with mdb-wt

      sajack@linux-p7yj /home/sajack/support-tools/mdb-wt git master () % python mdb-wt.py -p ~/mongo/corruption/db/collection-0--6386865262002950881.wt
=== mdb-wt.py -p /home/sajack/mongo/corruption/db/collection-0--6386865262002950881.wt
00000000: block_desc magic=120897(OK) major=1 minor=0 cksum=0xb72308d8
00001000: page recno=0 gen=1 msz=0x48 entries=2 type=7(ROW_LEAF) flags=0x4(no0) 
0000101c: block sz=0x1000 cksum=0x21a6bdaf flags=0x1(cksum)
00002000: page recno=0 gen=2 msz=0x33 entries=2 type=6(ROW_INT) flags=0x0() 
0000201c: block sz=0x1000 cksum=0x60d4ee05 flags=0x1(cksum)
00003000: page recno=0 gen=0 msz=0x32 entries=10 type=1(BLOCK_MANAGER) flags=0x0() 
0000301c: block sz=0x1000 cksum=0xf2e0bde0 flags=0x1(cksum)

      4) Corrupt the database (attached as corrupted.tar.bz2)
      Open ~/mongo/corruption/db/collection-0--6386865262002950881.wt with a hex editor
      Change hex value 0x61 at 0x1044 to 0x62
      Save

      5) Run mongod --repair

      sajack@linux-p7yj /home/sajack/mongo git master () % ./mongod --directoryperdb --dbpath corruption --storageEngine wiredtiger --repair
      Show
      1) Create the uncorrupted database(attached as uncorrupted.tar.bz2): ./mongod --directoryperdb --dbpath corruption --storageEngine wiredtiger > use db switched to db db > db.collection.insert({data: true }) 2) Set up database db for mdb-wt: sajack@linux-p7yj /home/sajack/mongo git master () % cd corruption/db sajack@linux-p7yj /home/sajack/mongo/corruption/db git master () % ls collection-0--6386865262002950881.wt index-1--6386865262002950881.wt sajack@linux-p7yj /home/sajack/mongo/corruption/db git master () % ln -s ../WiredTiger.wt . sajack@linux-p7yj /home/sajack/mongo/corruption/db git master () % ln -s ../WiredTiger.turtle . 3) Find offsets with mdb-wt sajack@linux-p7yj /home/sajack/support-tools/mdb-wt git master () % python mdb-wt.py -p ~/mongo/corruption/db/collection-0--6386865262002950881.wt === mdb-wt.py -p /home/sajack/mongo/corruption/db/collection-0--6386865262002950881.wt 00000000: block_desc magic=120897(OK) major=1 minor=0 cksum=0xb72308d8 00001000: page recno=0 gen=1 msz=0x48 entries=2 type=7(ROW_LEAF) flags=0x4(no0) 0000101c: block sz=0x1000 cksum=0x21a6bdaf flags=0x1(cksum) 00002000: page recno=0 gen=2 msz=0x33 entries=2 type=6(ROW_INT) flags=0x0() 0000201c: block sz=0x1000 cksum=0x60d4ee05 flags=0x1(cksum) 00003000: page recno=0 gen=0 msz=0x32 entries=10 type=1(BLOCK_MANAGER) flags=0x0() 0000301c: block sz=0x1000 cksum=0xf2e0bde0 flags=0x1(cksum) 4) Corrupt the database (attached as corrupted.tar.bz2) Open ~/mongo/corruption/db/collection-0--6386865262002950881.wt with a hex editor Change hex value 0x61 at 0x1044 to 0x62 Save 5) Run mongod --repair sajack@linux-p7yj /home/sajack/mongo git master () % ./mongod --directoryperdb --dbpath corruption --storageEngine wiredtiger --repair

      Manually flipping a bit in a WiredTiger database causes a mongod using it to fail to turn on. Executing mongod again with --repair will produce the same error.

      sajack@linux-p7yj /home/sajack/mongo git master () % ./mongod --directoryperdb --dbpath corruption --storageEngine wiredtiger                                                                                                    14 :(
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] MongoDB starting : pid=27278 port=27017 dbpath=corruption 64-bit host=linux-p7yj.site
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] _DEBUG build (which is slower)
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] 
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] 
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] db version v2.8.0-rc3
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] git version: bfaa98729f41d02a0aafe1efc44b2074e3d7ba14 modules: subscription
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1j 15 Oct 2014
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] build info: Linux linux-p7yj.site 3.17.4-2.g2d23787-desktop #1 SMP PREEMPT Tue Nov 25 20:17:30 UTC 2014 (2d23787) x86_64 BOOST_LIB_VERSION=1_49
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] allocator: tcmalloc
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] options: { storage: { dbPath: "corruption", directoryPerDB: true, engine: "wiredtiger", wiredTiger: { engineConfig: { checkpointDelaySecs: 60 } } } }
2014-12-18T15:22:41.056-0500 W STORAGE  [initandlisten] Detected configuration for non-active storage engine wiredTiger when current storage engine is wiredtiger
2014-12-18T15:22:41.056-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7G,session_max=20000,extensions=[local=(entry=index_collator_extension)],statistics=(all),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2014-12-18T15:22:41.671-0500 E STORAGE  [initandlisten] WiredTiger (0) [1418934161:671916][27278:0x7f4cacb92c80], file:db/collection-0--6386865262002950881.wt, cursor.prev: read checksum error [4096B @ 4096, 564575663 != 2778724596]
2014-12-18T15:22:41.671-0500 E STORAGE  [initandlisten] WiredTiger (0) [1418934161:671960][27278:0x7f4cacb92c80], file:db/collection-0--6386865262002950881.wt, cursor.prev: db/collection-0--6386865262002950881.wt: encountered an illegal file format or internal value
2014-12-18T15:22:41.671-0500 E STORAGE  [initandlisten] WiredTiger (-31804) [1418934161:671975][27278:0x7f4cacb92c80], file:db/collection-0--6386865262002950881.wt, cursor.prev: the process must exit and restart: WT_PANIC: WiredTiger library panic
2014-12-18T15:22:41.671-0500 I -        [initandlisten] Fatal Assertion 28558
2014-12-18T15:22:41.697-0500 I CONTROL  [initandlisten] 
 0x199c053 0x192a511 0x1912df6 0x10220c4 0x173b46c 0x216be7b 0x216c1d5 0x216cb9b 0x216cc31 0x207bdf4 0x207c0f9 0x20965af 0x209d87b 0x2099f67 0x20b670e 0x20b6162 0x208720b 0x20e158f 0x174671c 0x17465f2 0x1744cc5 0x1740ad9 0x173d36b 0x1699a87 0x169f38f 0x173a4f2 0x13339c7 0xfbb16b 0xfba0c3 0xfbc243 0xfbbeba 0x7f4ca98d0be5 0xfb9fe9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"159C053"},{"b":"400000","o":"152A511"},{"b":"400000","o":"1512DF6"},{"b":"400000","o":"C220C4"},{"b":"400000","o":"133B46C"},{"b":"400000","o":"1D6BE7B"},{"b":"400000","o":"1D6C1D5"},{"b":"400000","o":"1D6CB9B"},{"b":"400000","o":"1D6CC31"},{"b":"400000","o":"1C7BDF4"},{"b":"400000","o":"1C7C0F9"},{"b":"400000","o":"1C965AF"},{"b":"400000","o":"1C9D87B"},{"b":"400000","o":"1C99F67"},{"b":"400000","o":"1CB670E"},{"b":"400000","o":"1CB6162"},{"b":"400000","o":"1C8720B"},{"b":"400000","o":"1CE158F"},{"b":"400000","o":"134671C"},{"b":"400000","o":"13465F2"},{"b":"400000","o":"1344CC5"},{"b":"400000","o":"1340AD9"},{"b":"400000","o":"133D36B"},{"b":"400000","o":"1299A87"},{"b":"400000","o":"129F38F"},{"b":"400000","o":"133A4F2"},{"b":"400000","o":"F339C7"},{"b":"400000","o":"BBB16B"},{"b":"400000","o":"BBA0C3"},{"b":"400000","o":"BBC243"},{"b":"400000","o":"BBBEBA"},{"b":"7F4CA98AF000","o":"21BE5"},{"b":"400000","o":"BB9FE9"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc3", "gitVersion" : "bfaa98729f41d02a0aafe1efc44b2074e3d7ba14 modules: subscription", "uname" : { "sysname" : "Linux", "release" : "3.17.4-2.g2d23787-desktop", "version" : "#1 SMP PREEMPT Tue Nov 25 20:17:30 UTC 2014 (2d23787)", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "DCCBD4A18B46D3F97CB6C82AC62A85E1442E4EE5" }, { "b" : "7FFF6D0C8000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "6B2E43D38205BED52DFAD1887AC63806A7345757" }, { "b" : "7F4CAC78C000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "BE3019C7B6BB3BB0312613871673857E98EBF34E" }, { "b" : "7F4CAC546000", "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "4500EFC6B03F2515A34C8CBFFEB536EAB625BFFA" }, { "b" : "7F4CAC08E000", "path" : "/usr/lib64/libnetsnmpmibs.so.30", "elfType" : 3, "buildId" : "6BAB066098313A5B6F6E503B3E5CA738EC00A3EF" }, { "b" : "7F4CABE7F000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "FB719500964D1833D52CE8DD730849E89D96BA05" }, { "b" : "7F4CABC7B000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "666913EC064C25FDE1720DB77B1F57BDEB5D9B3A" }, { "b" : "7F4CAB8A7000", "path" : "/usr/lib64/librpm.so.3", "elfType" : 3, "buildId" : "038922E310A8B00BE9E9BA564CDAA47C87A184AA" }, { "b" : "7F4CAB656000", "path" : "/usr/lib64/librpmio.so.3", "elfType" : 3, "buildId" : "B42C339BCC9D9CB51416716F080F1A32013C92BC" }, { "b" : "7F4CAB3E7000", "path" : "/usr/lib64/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "AB25E6A40B2BFE5F7EE213E271A496F7724A5821" }, { "b" : "7F4CAB1DC000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "1B986139DE9922CBAD77FD86BBBEDD752844FEAF" }, { "b" : "7F4CAAEF9000", "path" : "/usr/lib64/libnetsnmp.so.30", "elfType" : 3, "buildId" : "B7518EBA89C271B322A57A42E3427E52AF54EB74" }, { "b" : "7F4CAAB0D000", "path" : "/lib64/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "D781931A27CC80C55A27B8448944BA7F5B0C3E08" }, { "b" : "7F4CAA80A000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "35D35D9CE781BE3A140A34242D998498615B021F" }, { "b" : "7F4CAA5EC000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "24A941B6C85E244BD2E4906928DC17AB63B0A38A" }, { "b" : "7F4CAA384000", "path" : "/lib64/libssl.so.1.0.0", "elfType" : 3, "buildId" : "44F5AFDAB585C00E3E94A89882CBFDB9BAA60F06" }, { "b" : "7F4CAA17C000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "0913D5EA4659CFA3FFAD451ACC84A6ED62772141" }, { "b" : "7F4CA9E74000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "893844CE9810835B76559F6EC3DC5A15E0D32A00" }, { "b" : "7F4CA9C5D000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "26253BEC1076626F73B8B13BED9510AE626E1EAD" }, { "b" : "7F4CA98AF000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "71DBBED0200919BA751AD20A545B9266258CD5E5" }, { "b" : "7F4CAC9A9000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "AFA98667969782208459E394F8C8F87AC7510710" }, { "b" : "7F4CA95E0000", "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "43FCBEA839B52BDA91D5CC00AFA9EC4AE0E324DC" }, { "b" : "7F4CA93AD000", "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "3EA89068D8F8FEE53C767950C64293B4D678FF61" }, { "b" : "7F4CA91A9000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "94324D6C48D39D04069706D01E6AF2D5D4429385" }, { "b" : "7F4CA8F9C000", "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "BC48F2A6DC84F7475BCD61857A742D8F2FE27139" }, { "b" : "7F4CA8D90000", "path" : "/usr/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "006D8E514AC130530294E6CFF5D8F77900DF869B" }, { "b" : "7F4CA8B6C000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E2BBA1E6E7493C181AB4DA9F2C771C8DB0875872" }, { "b" : "7F4CA8967000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "6F5E67B736B349EBE34CC2196BD4342736278777" }, { "b" : "7F4CA875E000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "684353207FA9BBA104A505EA9577DD10D74B9B51" }, { "b" : "7F4CA852E000", "path" : "/usr/lib64/liblua.so.5.1", "elfType" : 3, "buildId" : "8A9137F7CB106BDC6A51591F03AADABADB2088F9" }, { "b" : "7F4CA831F000", "path" : "/usr/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "0B6B84C801AD3C496B90FDA2D310259CEF812099" }, { "b" : "7F4CA8109000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "9E815EF41BF59451F1855B91195B2124E2A95F9E" }, { "b" : "7F4CA7EF3000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "984C530D8F881257D517CE19FCD80841F2230D80" }, { "b" : "7F4CA7CCD000", "path" : "/usr/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "8F2CBB2A0E341BEF4D5116EEDE8A83465DE6CA4B" }, { "b" : "7F4CA793A000", "path" : "/usr/lib/perl5/5.18.1/x86_64-linux-thread-multi/CORE/libperl.so", "elfType" : 3, "buildId" : "217B4A7EA7DF2A7B46BC93993FD7630EC1175D0C" }, { "b" : "7F4CA7736000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8EEA8B23F6DF125C66129A2DB29614A38300B77A" }, { "b" : "7F4CA751F000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "799562EB36D9479DF11B4BB13D8C29D852484FE8" }, { "b" : "7F4CA72B9000", "path" : "/usr/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "EF846533CF9AA9F40966FF8F2C1D120F1847BB45" }, { "b" : "7F4CA70B4000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "B6C7F251CBA059292F8892B74F4D2C167665C785" }, { "b" : "7F4CA6E79000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "355B60D4EFA51BB76FF03EB56A0986F555755453" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x23) [0x199c053]
 mongod(_ZN5mongo10logContextEPKc+0xC1) [0x192a511]
 mongod(_ZN5mongo13fassertFailedEi+0x86) [0x1912df6]
 mongod(_ZN5mongo7fassertEib+0x34) [0x10220c4]
 mongod(+0x133B46C) [0x173b46c]
 mongod(__wt_eventv+0x8AB) [0x216be7b]
 mongod(__wt_err+0x165) [0x216c1d5]
 mongod(__wt_panic+0x3B) [0x216cb9b]
 mongod(__wt_illegal_value+0x81) [0x216cc31]
 mongod(__wt_block_read_off+0x264) [0x207bdf4]
 mongod(__wt_bm_read+0x1D9) [0x207c0f9]
 mongod(__wt_bt_read+0x10F) [0x20965af]
 mongod(__wt_cache_read+0x14B) [0x209d87b]
 mongod(__wt_page_in_func+0xB7) [0x2099f67]
 mongod(+0x1CB670E) [0x20b670e]
 mongod(__wt_tree_walk+0x5C2) [0x20b6162]
 mongod(__wt_btcur_prev+0x49B) [0x208720b]
 mongod(+0x1CE158F) [0x20e158f]
 mongod(_ZN5mongo21WiredTigerRecordStore8Iterator7_locateERKNS_8RecordIdEb+0xCC) [0x174671c]
 mongod(_ZN5mongo21WiredTigerRecordStore8IteratorC2ERKS0_PNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionEb+0x172) [0x17465f2]
 mongod(_ZNK5mongo21WiredTigerRecordStore11getIteratorEPNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionE+0xD5) [0x1744cc5]
 mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextERKNS_10StringDataES5_bllPNS_28CappedDocumentDeleteCallbackEPNS_20WiredTigerSizeStorerE+0x499) [0x1740ad9]
 mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextERKNS_10StringDataES5_RKNS_17CollectionOptionsE+0x2EB) [0x173d36b]
 mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSs+0x157) [0x1699a87]
 mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x95F) [0x169f38f]
 mongod(+0x133A4F2) [0x173a4f2]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x1B7) [0x13339c7]
 mongod(+0xBBB16B) [0xfbb16b]
 mongod(_ZN5mongo13initAndListenEi+0x13) [0xfba0c3]
 mongod(+0xBBC243) [0xfbc243]
 mongod(main+0x2A) [0xfbbeba]
 libc.so.6(__libc_start_main+0xF5) [0x7f4ca98d0be5]
 mongod(+0xBB9FE9) [0xfb9fe9]
-----  END BACKTRACE  -----
2014-12-18T15:22:41.698-0500 I -        [initandlisten] 

***aborting after fassert() failure

      sajack@linux-p7yj /home/sajack/mongo git master () % ./mongod --directoryperdb --dbpath corruption --storageEngine wiredtiger --repair                                                                                           14 :(
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] MongoDB starting : pid=27291 port=27017 dbpath=corruption 64-bit host=linux-p7yj.site
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] _DEBUG build (which is slower)
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] 
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] 
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] db version v2.8.0-rc3
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] git version: bfaa98729f41d02a0aafe1efc44b2074e3d7ba14 modules: subscription
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1j 15 Oct 2014
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] build info: Linux linux-p7yj.site 3.17.4-2.g2d23787-desktop #1 SMP PREEMPT Tue Nov 25 20:17:30 UTC 2014 (2d23787) x86_64 BOOST_LIB_VERSION=1_49
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] allocator: tcmalloc
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "corruption", directoryPerDB: true, engine: "wiredtiger", wiredTiger: { engineConfig: { checkpointDelaySecs: 60 } } } }
2014-12-18T15:22:44.815-0500 W STORAGE  [initandlisten] Detected configuration for non-active storage engine wiredTiger when current storage engine is wiredtiger
2014-12-18T15:22:44.815-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7G,session_max=20000,extensions=[local=(entry=index_collator_extension)],statistics=(all),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2014-12-18T15:22:44.880-0500 I STORAGE  [initandlisten] Repairing size cache
2014-12-18T15:22:44.880-0500 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2014-12-18T15:22:44.880-0500 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2014-12-18T15:22:44.881-0500 I STORAGE  [initandlisten] Repairing catalog metadata
2014-12-18T15:22:44.882-0500 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2014-12-18T15:22:44.882-0500 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2014-12-18T15:22:44.883-0500 E STORAGE  [initandlisten] WiredTiger (0) [1418934164:883933][27291:0x7f34ac33dc80], file:db/collection-0--6386865262002950881.wt, cursor.prev: read checksum error [4096B @ 4096, 564575663 != 2778724596]
2014-12-18T15:22:44.883-0500 E STORAGE  [initandlisten] WiredTiger (0) [1418934164:883982][27291:0x7f34ac33dc80], file:db/collection-0--6386865262002950881.wt, cursor.prev: db/collection-0--6386865262002950881.wt: encountered an illegal file format or internal value
2014-12-18T15:22:44.884-0500 E STORAGE  [initandlisten] WiredTiger (-31804) [1418934164:884013][27291:0x7f34ac33dc80], file:db/collection-0--6386865262002950881.wt, cursor.prev: the process must exit and restart: WT_PANIC: WiredTiger library panic
2014-12-18T15:22:44.884-0500 I -        [initandlisten] Fatal Assertion 28558
2014-12-18T15:22:44.911-0500 I CONTROL  [initandlisten] 
 0x199c053 0x192a511 0x1912df6 0x10220c4 0x173b46c 0x216be7b 0x216c1d5 0x216cb9b 0x216cc31 0x207bdf4 0x207c0f9 0x20965af 0x209d87b 0x2099f67 0x20b670e 0x20b6162 0x208720b 0x20e158f 0x174671c 0x17465f2 0x1744cc5 0x1740ad9 0x173d36b 0x1699a87 0x169f38f 0x173a4f2 0x13339c7 0xfbb16b 0xfba0c3 0xfbc243 0xfbbeba 0x7f34a907bbe5 0xfb9fe9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"159C053"},{"b":"400000","o":"152A511"},{"b":"400000","o":"1512DF6"},{"b":"400000","o":"C220C4"},{"b":"400000","o":"133B46C"},{"b":"400000","o":"1D6BE7B"},{"b":"400000","o":"1D6C1D5"},{"b":"400000","o":"1D6CB9B"},{"b":"400000","o":"1D6CC31"},{"b":"400000","o":"1C7BDF4"},{"b":"400000","o":"1C7C0F9"},{"b":"400000","o":"1C965AF"},{"b":"400000","o":"1C9D87B"},{"b":"400000","o":"1C99F67"},{"b":"400000","o":"1CB670E"},{"b":"400000","o":"1CB6162"},{"b":"400000","o":"1C8720B"},{"b":"400000","o":"1CE158F"},{"b":"400000","o":"134671C"},{"b":"400000","o":"13465F2"},{"b":"400000","o":"1344CC5"},{"b":"400000","o":"1340AD9"},{"b":"400000","o":"133D36B"},{"b":"400000","o":"1299A87"},{"b":"400000","o":"129F38F"},{"b":"400000","o":"133A4F2"},{"b":"400000","o":"F339C7"},{"b":"400000","o":"BBB16B"},{"b":"400000","o":"BBA0C3"},{"b":"400000","o":"BBC243"},{"b":"400000","o":"BBBEBA"},{"b":"7F34A905A000","o":"21BE5"},{"b":"400000","o":"BB9FE9"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc3", "gitVersion" : "bfaa98729f41d02a0aafe1efc44b2074e3d7ba14 modules: subscription", "uname" : { "sysname" : "Linux", "release" : "3.17.4-2.g2d23787-desktop", "version" : "#1 SMP PREEMPT Tue Nov 25 20:17:30 UTC 2014 (2d23787)", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "DCCBD4A18B46D3F97CB6C82AC62A85E1442E4EE5" }, { "b" : "7FFFB35FE000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "6B2E43D38205BED52DFAD1887AC63806A7345757" }, { "b" : "7F34ABF37000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "BE3019C7B6BB3BB0312613871673857E98EBF34E" }, { "b" : "7F34ABCF1000", "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "4500EFC6B03F2515A34C8CBFFEB536EAB625BFFA" }, { "b" : "7F34AB839000", "path" : "/usr/lib64/libnetsnmpmibs.so.30", "elfType" : 3, "buildId" : "6BAB066098313A5B6F6E503B3E5CA738EC00A3EF" }, { "b" : "7F34AB62A000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "FB719500964D1833D52CE8DD730849E89D96BA05" }, { "b" : "7F34AB426000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "666913EC064C25FDE1720DB77B1F57BDEB5D9B3A" }, { "b" : "7F34AB052000", "path" : "/usr/lib64/librpm.so.3", "elfType" : 3, "buildId" : "038922E310A8B00BE9E9BA564CDAA47C87A184AA" }, { "b" : "7F34AAE01000", "path" : "/usr/lib64/librpmio.so.3", "elfType" : 3, "buildId" : "B42C339BCC9D9CB51416716F080F1A32013C92BC" }, { "b" : "7F34AAB92000", "path" : "/usr/lib64/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "AB25E6A40B2BFE5F7EE213E271A496F7724A5821" }, { "b" : "7F34AA987000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "1B986139DE9922CBAD77FD86BBBEDD752844FEAF" }, { "b" : "7F34AA6A4000", "path" : "/usr/lib64/libnetsnmp.so.30", "elfType" : 3, "buildId" : "B7518EBA89C271B322A57A42E3427E52AF54EB74" }, { "b" : "7F34AA2B8000", "path" : "/lib64/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "D781931A27CC80C55A27B8448944BA7F5B0C3E08" }, { "b" : "7F34A9FB5000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "35D35D9CE781BE3A140A34242D998498615B021F" }, { "b" : "7F34A9D97000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "24A941B6C85E244BD2E4906928DC17AB63B0A38A" }, { "b" : "7F34A9B2F000", "path" : "/lib64/libssl.so.1.0.0", "elfType" : 3, "buildId" : "44F5AFDAB585C00E3E94A89882CBFDB9BAA60F06" }, { "b" : "7F34A9927000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "0913D5EA4659CFA3FFAD451ACC84A6ED62772141" }, { "b" : "7F34A961F000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "893844CE9810835B76559F6EC3DC5A15E0D32A00" }, { "b" : "7F34A9408000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "26253BEC1076626F73B8B13BED9510AE626E1EAD" }, { "b" : "7F34A905A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "71DBBED0200919BA751AD20A545B9266258CD5E5" }, { "b" : "7F34AC154000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "AFA98667969782208459E394F8C8F87AC7510710" }, { "b" : "7F34A8D8B000", "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "43FCBEA839B52BDA91D5CC00AFA9EC4AE0E324DC" }, { "b" : "7F34A8B58000", "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "3EA89068D8F8FEE53C767950C64293B4D678FF61" }, { "b" : "7F34A8954000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "94324D6C48D39D04069706D01E6AF2D5D4429385" }, { "b" : "7F34A8747000", "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "BC48F2A6DC84F7475BCD61857A742D8F2FE27139" }, { "b" : "7F34A853B000", "path" : "/usr/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "006D8E514AC130530294E6CFF5D8F77900DF869B" }, { "b" : "7F34A8317000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E2BBA1E6E7493C181AB4DA9F2C771C8DB0875872" }, { "b" : "7F34A8112000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "6F5E67B736B349EBE34CC2196BD4342736278777" }, { "b" : "7F34A7F09000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "684353207FA9BBA104A505EA9577DD10D74B9B51" }, { "b" : "7F34A7CD9000", "path" : "/usr/lib64/liblua.so.5.1", "elfType" : 3, "buildId" : "8A9137F7CB106BDC6A51591F03AADABADB2088F9" }, { "b" : "7F34A7ACA000", "path" : "/usr/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "0B6B84C801AD3C496B90FDA2D310259CEF812099" }, { "b" : "7F34A78B4000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "9E815EF41BF59451F1855B91195B2124E2A95F9E" }, { "b" : "7F34A769E000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "984C530D8F881257D517CE19FCD80841F2230D80" }, { "b" : "7F34A7478000", "path" : "/usr/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "8F2CBB2A0E341BEF4D5116EEDE8A83465DE6CA4B" }, { "b" : "7F34A70E5000", "path" : "/usr/lib/perl5/5.18.1/x86_64-linux-thread-multi/CORE/libperl.so", "elfType" : 3, "buildId" : "217B4A7EA7DF2A7B46BC93993FD7630EC1175D0C" }, { "b" : "7F34A6EE1000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8EEA8B23F6DF125C66129A2DB29614A38300B77A" }, { "b" : "7F34A6CCA000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "799562EB36D9479DF11B4BB13D8C29D852484FE8" }, { "b" : "7F34A6A64000", "path" : "/usr/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "EF846533CF9AA9F40966FF8F2C1D120F1847BB45" }, { "b" : "7F34A685F000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "B6C7F251CBA059292F8892B74F4D2C167665C785" }, { "b" : "7F34A6624000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "355B60D4EFA51BB76FF03EB56A0986F555755453" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x23) [0x199c053]
 mongod(_ZN5mongo10logContextEPKc+0xC1) [0x192a511]
 mongod(_ZN5mongo13fassertFailedEi+0x86) [0x1912df6]
 mongod(_ZN5mongo7fassertEib+0x34) [0x10220c4]
 mongod(+0x133B46C) [0x173b46c]
 mongod(__wt_eventv+0x8AB) [0x216be7b]
 mongod(__wt_err+0x165) [0x216c1d5]
 mongod(__wt_panic+0x3B) [0x216cb9b]
 mongod(__wt_illegal_value+0x81) [0x216cc31]
 mongod(__wt_block_read_off+0x264) [0x207bdf4]
 mongod(__wt_bm_read+0x1D9) [0x207c0f9]
 mongod(__wt_bt_read+0x10F) [0x20965af]
 mongod(__wt_cache_read+0x14B) [0x209d87b]
 mongod(__wt_page_in_func+0xB7) [0x2099f67]
 mongod(+0x1CB670E) [0x20b670e]
 mongod(__wt_tree_walk+0x5C2) [0x20b6162]
 mongod(__wt_btcur_prev+0x49B) [0x208720b]
 mongod(+0x1CE158F) [0x20e158f]
 mongod(_ZN5mongo21WiredTigerRecordStore8Iterator7_locateERKNS_8RecordIdEb+0xCC) [0x174671c]
 mongod(_ZN5mongo21WiredTigerRecordStore8IteratorC2ERKS0_PNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionEb+0x172) [0x17465f2]
 mongod(_ZNK5mongo21WiredTigerRecordStore11getIteratorEPNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionE+0xD5) [0x1744cc5]
 mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextERKNS_10StringDataES5_bllPNS_28CappedDocumentDeleteCallbackEPNS_20WiredTigerSizeStorerE+0x499) [0x1740ad9]
 mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextERKNS_10StringDataES5_RKNS_17CollectionOptionsE+0x2EB) [0x173d36b]
 mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSs+0x157) [0x1699a87]
 mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x95F) [0x169f38f]
 mongod(+0x133A4F2) [0x173a4f2]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x1B7) [0x13339c7]
 mongod(+0xBBB16B) [0xfbb16b]
 mongod(_ZN5mongo13initAndListenEi+0x13) [0xfba0c3]
 mongod(+0xBBC243) [0xfbc243]
 mongod(main+0x2A) [0xfbbeba]
 libc.so.6(__libc_start_main+0xF5) [0x7f34a907bbe5]
 mongod(+0xBB9FE9) [0xfb9fe9]
-----  END BACKTRACE  -----
2014-12-18T15:22:44.911-0500 I -        [initandlisten] 

***aborting after fassert() failure

        1. corrupted.tar.bz2
          5 kB
        2. uncorrupted.tar.bz2
          7 kB

            Assignee:
            Unassigned Unassigned
            Reporter:
            spencer.jackson@mongodb.com Spencer Jackson
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: