[SERVER-16596] Repairing a WiredTiger database with a flipped bit triggers a checksum fassert Created: 18/Dec/14  Updated: 24/Jan/15  Resolved: 12/Jan/15

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Spencer Jackson Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: 28qa, wiredtiger
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File corrupted.tar.bz2     File uncorrupted.tar.bz2    
Issue Links:
Duplicate
duplicates SERVER-16172 mongod --repair terminates before rep... Closed
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

1) Create the uncorrupted database(attached as uncorrupted.tar.bz2):

./mongod --directoryperdb --dbpath corruption --storageEngine wiredtiger
> use db
switched to db db
> db.collection.insert({data: true})

2) Set up database db for mdb-wt:

sajack@linux-p7yj /home/sajack/mongo git master () % cd corruption/db
sajack@linux-p7yj /home/sajack/mongo/corruption/db git master () % ls
collection-0--6386865262002950881.wt  index-1--6386865262002950881.wt
sajack@linux-p7yj /home/sajack/mongo/corruption/db git master () % ln -s ../WiredTiger.wt .
sajack@linux-p7yj /home/sajack/mongo/corruption/db git master () % ln -s ../WiredTiger.turtle .

3) Find offsets with mdb-wt

sajack@linux-p7yj /home/sajack/support-tools/mdb-wt git master () % python mdb-wt.py -p ~/mongo/corruption/db/collection-0--6386865262002950881.wt
=== mdb-wt.py -p /home/sajack/mongo/corruption/db/collection-0--6386865262002950881.wt
00000000: block_desc magic=120897(OK) major=1 minor=0 cksum=0xb72308d8
00001000: page recno=0 gen=1 msz=0x48 entries=2 type=7(ROW_LEAF) flags=0x4(no0) 
0000101c: block sz=0x1000 cksum=0x21a6bdaf flags=0x1(cksum)
00002000: page recno=0 gen=2 msz=0x33 entries=2 type=6(ROW_INT) flags=0x0() 
0000201c: block sz=0x1000 cksum=0x60d4ee05 flags=0x1(cksum)
00003000: page recno=0 gen=0 msz=0x32 entries=10 type=1(BLOCK_MANAGER) flags=0x0() 
0000301c: block sz=0x1000 cksum=0xf2e0bde0 flags=0x1(cksum)

4) Corrupt the database (attached as corrupted.tar.bz2)
Open ~/mongo/corruption/db/collection-0--6386865262002950881.wt with a hex editor
Change hex value 0x61 at 0x1044 to 0x62
Save

5) Run mongod --repair

sajack@linux-p7yj /home/sajack/mongo git master () % ./mongod --directoryperdb --dbpath corruption --storageEngine wiredtiger --repair

Participants:

 Description   

Manually flipping a bit in a WiredTiger database causes a mongod using it to fail to turn on. Executing mongod again with --repair will produce the same error.

sajack@linux-p7yj /home/sajack/mongo git master () % ./mongod --directoryperdb --dbpath corruption --storageEngine wiredtiger                                                                                                    14 :(
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] MongoDB starting : pid=27278 port=27017 dbpath=corruption 64-bit host=linux-p7yj.site
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] _DEBUG build (which is slower)
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] 
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] 
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] db version v2.8.0-rc3
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] git version: bfaa98729f41d02a0aafe1efc44b2074e3d7ba14 modules: subscription
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1j 15 Oct 2014
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] build info: Linux linux-p7yj.site 3.17.4-2.g2d23787-desktop #1 SMP PREEMPT Tue Nov 25 20:17:30 UTC 2014 (2d23787) x86_64 BOOST_LIB_VERSION=1_49
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] allocator: tcmalloc
2014-12-18T15:22:41.055-0500 I CONTROL  [initandlisten] options: { storage: { dbPath: "corruption", directoryPerDB: true, engine: "wiredtiger", wiredTiger: { engineConfig: { checkpointDelaySecs: 60 } } } }
2014-12-18T15:22:41.056-0500 W STORAGE  [initandlisten] Detected configuration for non-active storage engine wiredTiger when current storage engine is wiredtiger
2014-12-18T15:22:41.056-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7G,session_max=20000,extensions=[local=(entry=index_collator_extension)],statistics=(all),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2014-12-18T15:22:41.671-0500 E STORAGE  [initandlisten] WiredTiger (0) [1418934161:671916][27278:0x7f4cacb92c80], file:db/collection-0--6386865262002950881.wt, cursor.prev: read checksum error [4096B @ 4096, 564575663 != 2778724596]
2014-12-18T15:22:41.671-0500 E STORAGE  [initandlisten] WiredTiger (0) [1418934161:671960][27278:0x7f4cacb92c80], file:db/collection-0--6386865262002950881.wt, cursor.prev: db/collection-0--6386865262002950881.wt: encountered an illegal file format or internal value
2014-12-18T15:22:41.671-0500 E STORAGE  [initandlisten] WiredTiger (-31804) [1418934161:671975][27278:0x7f4cacb92c80], file:db/collection-0--6386865262002950881.wt, cursor.prev: the process must exit and restart: WT_PANIC: WiredTiger library panic
2014-12-18T15:22:41.671-0500 I -        [initandlisten] Fatal Assertion 28558
2014-12-18T15:22:41.697-0500 I CONTROL  [initandlisten] 
 0x199c053 0x192a511 0x1912df6 0x10220c4 0x173b46c 0x216be7b 0x216c1d5 0x216cb9b 0x216cc31 0x207bdf4 0x207c0f9 0x20965af 0x209d87b 0x2099f67 0x20b670e 0x20b6162 0x208720b 0x20e158f 0x174671c 0x17465f2 0x1744cc5 0x1740ad9 0x173d36b 0x1699a87 0x169f38f 0x173a4f2 0x13339c7 0xfbb16b 0xfba0c3 0xfbc243 0xfbbeba 0x7f4ca98d0be5 0xfb9fe9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"159C053"},{"b":"400000","o":"152A511"},{"b":"400000","o":"1512DF6"},{"b":"400000","o":"C220C4"},{"b":"400000","o":"133B46C"},{"b":"400000","o":"1D6BE7B"},{"b":"400000","o":"1D6C1D5"},{"b":"400000","o":"1D6CB9B"},{"b":"400000","o":"1D6CC31"},{"b":"400000","o":"1C7BDF4"},{"b":"400000","o":"1C7C0F9"},{"b":"400000","o":"1C965AF"},{"b":"400000","o":"1C9D87B"},{"b":"400000","o":"1C99F67"},{"b":"400000","o":"1CB670E"},{"b":"400000","o":"1CB6162"},{"b":"400000","o":"1C8720B"},{"b":"400000","o":"1CE158F"},{"b":"400000","o":"134671C"},{"b":"400000","o":"13465F2"},{"b":"400000","o":"1344CC5"},{"b":"400000","o":"1340AD9"},{"b":"400000","o":"133D36B"},{"b":"400000","o":"1299A87"},{"b":"400000","o":"129F38F"},{"b":"400000","o":"133A4F2"},{"b":"400000","o":"F339C7"},{"b":"400000","o":"BBB16B"},{"b":"400000","o":"BBA0C3"},{"b":"400000","o":"BBC243"},{"b":"400000","o":"BBBEBA"},{"b":"7F4CA98AF000","o":"21BE5"},{"b":"400000","o":"BB9FE9"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc3", "gitVersion" : "bfaa98729f41d02a0aafe1efc44b2074e3d7ba14 modules: subscription", "uname" : { "sysname" : "Linux", "release" : "3.17.4-2.g2d23787-desktop", "version" : "#1 SMP PREEMPT Tue Nov 25 20:17:30 UTC 2014 (2d23787)", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "DCCBD4A18B46D3F97CB6C82AC62A85E1442E4EE5" }, { "b" : "7FFF6D0C8000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "6B2E43D38205BED52DFAD1887AC63806A7345757" }, { "b" : "7F4CAC78C000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "BE3019C7B6BB3BB0312613871673857E98EBF34E" }, { "b" : "7F4CAC546000", "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "4500EFC6B03F2515A34C8CBFFEB536EAB625BFFA" }, { "b" : "7F4CAC08E000", "path" : "/usr/lib64/libnetsnmpmibs.so.30", "elfType" : 3, "buildId" : "6BAB066098313A5B6F6E503B3E5CA738EC00A3EF" }, { "b" : "7F4CABE7F000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "FB719500964D1833D52CE8DD730849E89D96BA05" }, { "b" : "7F4CABC7B000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "666913EC064C25FDE1720DB77B1F57BDEB5D9B3A" }, { "b" : "7F4CAB8A7000", "path" : "/usr/lib64/librpm.so.3", "elfType" : 3, "buildId" : "038922E310A8B00BE9E9BA564CDAA47C87A184AA" }, { "b" : "7F4CAB656000", "path" : "/usr/lib64/librpmio.so.3", "elfType" : 3, "buildId" : "B42C339BCC9D9CB51416716F080F1A32013C92BC" }, { "b" : "7F4CAB3E7000", "path" : "/usr/lib64/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "AB25E6A40B2BFE5F7EE213E271A496F7724A5821" }, { "b" : "7F4CAB1DC000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "1B986139DE9922CBAD77FD86BBBEDD752844FEAF" }, { "b" : "7F4CAAEF9000", "path" : "/usr/lib64/libnetsnmp.so.30", "elfType" : 3, "buildId" : "B7518EBA89C271B322A57A42E3427E52AF54EB74" }, { "b" : "7F4CAAB0D000", "path" : "/lib64/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "D781931A27CC80C55A27B8448944BA7F5B0C3E08" }, { "b" : "7F4CAA80A000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "35D35D9CE781BE3A140A34242D998498615B021F" }, { "b" : "7F4CAA5EC000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "24A941B6C85E244BD2E4906928DC17AB63B0A38A" }, { "b" : "7F4CAA384000", "path" : "/lib64/libssl.so.1.0.0", "elfType" : 3, "buildId" : "44F5AFDAB585C00E3E94A89882CBFDB9BAA60F06" }, { "b" : "7F4CAA17C000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "0913D5EA4659CFA3FFAD451ACC84A6ED62772141" }, { "b" : "7F4CA9E74000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "893844CE9810835B76559F6EC3DC5A15E0D32A00" }, { "b" : "7F4CA9C5D000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "26253BEC1076626F73B8B13BED9510AE626E1EAD" }, { "b" : "7F4CA98AF000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "71DBBED0200919BA751AD20A545B9266258CD5E5" }, { "b" : "7F4CAC9A9000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "AFA98667969782208459E394F8C8F87AC7510710" }, { "b" : "7F4CA95E0000", "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "43FCBEA839B52BDA91D5CC00AFA9EC4AE0E324DC" }, { "b" : "7F4CA93AD000", "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "3EA89068D8F8FEE53C767950C64293B4D678FF61" }, { "b" : "7F4CA91A9000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "94324D6C48D39D04069706D01E6AF2D5D4429385" }, { "b" : "7F4CA8F9C000", "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "BC48F2A6DC84F7475BCD61857A742D8F2FE27139" }, { "b" : "7F4CA8D90000", "path" : "/usr/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "006D8E514AC130530294E6CFF5D8F77900DF869B" }, { "b" : "7F4CA8B6C000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E2BBA1E6E7493C181AB4DA9F2C771C8DB0875872" }, { "b" : "7F4CA8967000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "6F5E67B736B349EBE34CC2196BD4342736278777" }, { "b" : "7F4CA875E000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "684353207FA9BBA104A505EA9577DD10D74B9B51" }, { "b" : "7F4CA852E000", "path" : "/usr/lib64/liblua.so.5.1", "elfType" : 3, "buildId" : "8A9137F7CB106BDC6A51591F03AADABADB2088F9" }, { "b" : "7F4CA831F000", "path" : "/usr/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "0B6B84C801AD3C496B90FDA2D310259CEF812099" }, { "b" : "7F4CA8109000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "9E815EF41BF59451F1855B91195B2124E2A95F9E" }, { "b" : "7F4CA7EF3000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "984C530D8F881257D517CE19FCD80841F2230D80" }, { "b" : "7F4CA7CCD000", "path" : "/usr/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "8F2CBB2A0E341BEF4D5116EEDE8A83465DE6CA4B" }, { "b" : "7F4CA793A000", "path" : "/usr/lib/perl5/5.18.1/x86_64-linux-thread-multi/CORE/libperl.so", "elfType" : 3, "buildId" : "217B4A7EA7DF2A7B46BC93993FD7630EC1175D0C" }, { "b" : "7F4CA7736000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8EEA8B23F6DF125C66129A2DB29614A38300B77A" }, { "b" : "7F4CA751F000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "799562EB36D9479DF11B4BB13D8C29D852484FE8" }, { "b" : "7F4CA72B9000", "path" : "/usr/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "EF846533CF9AA9F40966FF8F2C1D120F1847BB45" }, { "b" : "7F4CA70B4000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "B6C7F251CBA059292F8892B74F4D2C167665C785" }, { "b" : "7F4CA6E79000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "355B60D4EFA51BB76FF03EB56A0986F555755453" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x23) [0x199c053]
 mongod(_ZN5mongo10logContextEPKc+0xC1) [0x192a511]
 mongod(_ZN5mongo13fassertFailedEi+0x86) [0x1912df6]
 mongod(_ZN5mongo7fassertEib+0x34) [0x10220c4]
 mongod(+0x133B46C) [0x173b46c]
 mongod(__wt_eventv+0x8AB) [0x216be7b]
 mongod(__wt_err+0x165) [0x216c1d5]
 mongod(__wt_panic+0x3B) [0x216cb9b]
 mongod(__wt_illegal_value+0x81) [0x216cc31]
 mongod(__wt_block_read_off+0x264) [0x207bdf4]
 mongod(__wt_bm_read+0x1D9) [0x207c0f9]
 mongod(__wt_bt_read+0x10F) [0x20965af]
 mongod(__wt_cache_read+0x14B) [0x209d87b]
 mongod(__wt_page_in_func+0xB7) [0x2099f67]
 mongod(+0x1CB670E) [0x20b670e]
 mongod(__wt_tree_walk+0x5C2) [0x20b6162]
 mongod(__wt_btcur_prev+0x49B) [0x208720b]
 mongod(+0x1CE158F) [0x20e158f]
 mongod(_ZN5mongo21WiredTigerRecordStore8Iterator7_locateERKNS_8RecordIdEb+0xCC) [0x174671c]
 mongod(_ZN5mongo21WiredTigerRecordStore8IteratorC2ERKS0_PNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionEb+0x172) [0x17465f2]
 mongod(_ZNK5mongo21WiredTigerRecordStore11getIteratorEPNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionE+0xD5) [0x1744cc5]
 mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextERKNS_10StringDataES5_bllPNS_28CappedDocumentDeleteCallbackEPNS_20WiredTigerSizeStorerE+0x499) [0x1740ad9]
 mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextERKNS_10StringDataES5_RKNS_17CollectionOptionsE+0x2EB) [0x173d36b]
 mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSs+0x157) [0x1699a87]
 mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x95F) [0x169f38f]
 mongod(+0x133A4F2) [0x173a4f2]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x1B7) [0x13339c7]
 mongod(+0xBBB16B) [0xfbb16b]
 mongod(_ZN5mongo13initAndListenEi+0x13) [0xfba0c3]
 mongod(+0xBBC243) [0xfbc243]
 mongod(main+0x2A) [0xfbbeba]
 libc.so.6(__libc_start_main+0xF5) [0x7f4ca98d0be5]
 mongod(+0xBB9FE9) [0xfb9fe9]
-----  END BACKTRACE  -----
2014-12-18T15:22:41.698-0500 I -        [initandlisten] 
 
***aborting after fassert() failure

sajack@linux-p7yj /home/sajack/mongo git master () % ./mongod --directoryperdb --dbpath corruption --storageEngine wiredtiger --repair                                                                                           14 :(
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] MongoDB starting : pid=27291 port=27017 dbpath=corruption 64-bit host=linux-p7yj.site
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] _DEBUG build (which is slower)
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] 
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] 
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] db version v2.8.0-rc3
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] git version: bfaa98729f41d02a0aafe1efc44b2074e3d7ba14 modules: subscription
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1j 15 Oct 2014
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] build info: Linux linux-p7yj.site 3.17.4-2.g2d23787-desktop #1 SMP PREEMPT Tue Nov 25 20:17:30 UTC 2014 (2d23787) x86_64 BOOST_LIB_VERSION=1_49
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] allocator: tcmalloc
2014-12-18T15:22:44.814-0500 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "corruption", directoryPerDB: true, engine: "wiredtiger", wiredTiger: { engineConfig: { checkpointDelaySecs: 60 } } } }
2014-12-18T15:22:44.815-0500 W STORAGE  [initandlisten] Detected configuration for non-active storage engine wiredTiger when current storage engine is wiredtiger
2014-12-18T15:22:44.815-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7G,session_max=20000,extensions=[local=(entry=index_collator_extension)],statistics=(all),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2014-12-18T15:22:44.880-0500 I STORAGE  [initandlisten] Repairing size cache
2014-12-18T15:22:44.880-0500 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2014-12-18T15:22:44.880-0500 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2014-12-18T15:22:44.881-0500 I STORAGE  [initandlisten] Repairing catalog metadata
2014-12-18T15:22:44.882-0500 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2014-12-18T15:22:44.882-0500 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2014-12-18T15:22:44.883-0500 E STORAGE  [initandlisten] WiredTiger (0) [1418934164:883933][27291:0x7f34ac33dc80], file:db/collection-0--6386865262002950881.wt, cursor.prev: read checksum error [4096B @ 4096, 564575663 != 2778724596]
2014-12-18T15:22:44.883-0500 E STORAGE  [initandlisten] WiredTiger (0) [1418934164:883982][27291:0x7f34ac33dc80], file:db/collection-0--6386865262002950881.wt, cursor.prev: db/collection-0--6386865262002950881.wt: encountered an illegal file format or internal value
2014-12-18T15:22:44.884-0500 E STORAGE  [initandlisten] WiredTiger (-31804) [1418934164:884013][27291:0x7f34ac33dc80], file:db/collection-0--6386865262002950881.wt, cursor.prev: the process must exit and restart: WT_PANIC: WiredTiger library panic
2014-12-18T15:22:44.884-0500 I -        [initandlisten] Fatal Assertion 28558
2014-12-18T15:22:44.911-0500 I CONTROL  [initandlisten] 
 0x199c053 0x192a511 0x1912df6 0x10220c4 0x173b46c 0x216be7b 0x216c1d5 0x216cb9b 0x216cc31 0x207bdf4 0x207c0f9 0x20965af 0x209d87b 0x2099f67 0x20b670e 0x20b6162 0x208720b 0x20e158f 0x174671c 0x17465f2 0x1744cc5 0x1740ad9 0x173d36b 0x1699a87 0x169f38f 0x173a4f2 0x13339c7 0xfbb16b 0xfba0c3 0xfbc243 0xfbbeba 0x7f34a907bbe5 0xfb9fe9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"159C053"},{"b":"400000","o":"152A511"},{"b":"400000","o":"1512DF6"},{"b":"400000","o":"C220C4"},{"b":"400000","o":"133B46C"},{"b":"400000","o":"1D6BE7B"},{"b":"400000","o":"1D6C1D5"},{"b":"400000","o":"1D6CB9B"},{"b":"400000","o":"1D6CC31"},{"b":"400000","o":"1C7BDF4"},{"b":"400000","o":"1C7C0F9"},{"b":"400000","o":"1C965AF"},{"b":"400000","o":"1C9D87B"},{"b":"400000","o":"1C99F67"},{"b":"400000","o":"1CB670E"},{"b":"400000","o":"1CB6162"},{"b":"400000","o":"1C8720B"},{"b":"400000","o":"1CE158F"},{"b":"400000","o":"134671C"},{"b":"400000","o":"13465F2"},{"b":"400000","o":"1344CC5"},{"b":"400000","o":"1340AD9"},{"b":"400000","o":"133D36B"},{"b":"400000","o":"1299A87"},{"b":"400000","o":"129F38F"},{"b":"400000","o":"133A4F2"},{"b":"400000","o":"F339C7"},{"b":"400000","o":"BBB16B"},{"b":"400000","o":"BBA0C3"},{"b":"400000","o":"BBC243"},{"b":"400000","o":"BBBEBA"},{"b":"7F34A905A000","o":"21BE5"},{"b":"400000","o":"BB9FE9"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc3", "gitVersion" : "bfaa98729f41d02a0aafe1efc44b2074e3d7ba14 modules: subscription", "uname" : { "sysname" : "Linux", "release" : "3.17.4-2.g2d23787-desktop", "version" : "#1 SMP PREEMPT Tue Nov 25 20:17:30 UTC 2014 (2d23787)", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "DCCBD4A18B46D3F97CB6C82AC62A85E1442E4EE5" }, { "b" : "7FFFB35FE000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "6B2E43D38205BED52DFAD1887AC63806A7345757" }, { "b" : "7F34ABF37000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "BE3019C7B6BB3BB0312613871673857E98EBF34E" }, { "b" : "7F34ABCF1000", "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "4500EFC6B03F2515A34C8CBFFEB536EAB625BFFA" }, { "b" : "7F34AB839000", "path" : "/usr/lib64/libnetsnmpmibs.so.30", "elfType" : 3, "buildId" : "6BAB066098313A5B6F6E503B3E5CA738EC00A3EF" }, { "b" : "7F34AB62A000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "FB719500964D1833D52CE8DD730849E89D96BA05" }, { "b" : "7F34AB426000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "666913EC064C25FDE1720DB77B1F57BDEB5D9B3A" }, { "b" : "7F34AB052000", "path" : "/usr/lib64/librpm.so.3", "elfType" : 3, "buildId" : "038922E310A8B00BE9E9BA564CDAA47C87A184AA" }, { "b" : "7F34AAE01000", "path" : "/usr/lib64/librpmio.so.3", "elfType" : 3, "buildId" : "B42C339BCC9D9CB51416716F080F1A32013C92BC" }, { "b" : "7F34AAB92000", "path" : "/usr/lib64/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "AB25E6A40B2BFE5F7EE213E271A496F7724A5821" }, { "b" : "7F34AA987000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "1B986139DE9922CBAD77FD86BBBEDD752844FEAF" }, { "b" : "7F34AA6A4000", "path" : "/usr/lib64/libnetsnmp.so.30", "elfType" : 3, "buildId" : "B7518EBA89C271B322A57A42E3427E52AF54EB74" }, { "b" : "7F34AA2B8000", "path" : "/lib64/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "D781931A27CC80C55A27B8448944BA7F5B0C3E08" }, { "b" : "7F34A9FB5000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "35D35D9CE781BE3A140A34242D998498615B021F" }, { "b" : "7F34A9D97000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "24A941B6C85E244BD2E4906928DC17AB63B0A38A" }, { "b" : "7F34A9B2F000", "path" : "/lib64/libssl.so.1.0.0", "elfType" : 3, "buildId" : "44F5AFDAB585C00E3E94A89882CBFDB9BAA60F06" }, { "b" : "7F34A9927000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "0913D5EA4659CFA3FFAD451ACC84A6ED62772141" }, { "b" : "7F34A961F000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "893844CE9810835B76559F6EC3DC5A15E0D32A00" }, { "b" : "7F34A9408000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "26253BEC1076626F73B8B13BED9510AE626E1EAD" }, { "b" : "7F34A905A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "71DBBED0200919BA751AD20A545B9266258CD5E5" }, { "b" : "7F34AC154000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "AFA98667969782208459E394F8C8F87AC7510710" }, { "b" : "7F34A8D8B000", "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "43FCBEA839B52BDA91D5CC00AFA9EC4AE0E324DC" }, { "b" : "7F34A8B58000", "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "3EA89068D8F8FEE53C767950C64293B4D678FF61" }, { "b" : "7F34A8954000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "94324D6C48D39D04069706D01E6AF2D5D4429385" }, { "b" : "7F34A8747000", "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "BC48F2A6DC84F7475BCD61857A742D8F2FE27139" }, { "b" : "7F34A853B000", "path" : "/usr/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "006D8E514AC130530294E6CFF5D8F77900DF869B" }, { "b" : "7F34A8317000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E2BBA1E6E7493C181AB4DA9F2C771C8DB0875872" }, { "b" : "7F34A8112000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "6F5E67B736B349EBE34CC2196BD4342736278777" }, { "b" : "7F34A7F09000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "684353207FA9BBA104A505EA9577DD10D74B9B51" }, { "b" : "7F34A7CD9000", "path" : "/usr/lib64/liblua.so.5.1", "elfType" : 3, "buildId" : "8A9137F7CB106BDC6A51591F03AADABADB2088F9" }, { "b" : "7F34A7ACA000", "path" : "/usr/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "0B6B84C801AD3C496B90FDA2D310259CEF812099" }, { "b" : "7F34A78B4000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "9E815EF41BF59451F1855B91195B2124E2A95F9E" }, { "b" : "7F34A769E000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "984C530D8F881257D517CE19FCD80841F2230D80" }, { "b" : "7F34A7478000", "path" : "/usr/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "8F2CBB2A0E341BEF4D5116EEDE8A83465DE6CA4B" }, { "b" : "7F34A70E5000", "path" : "/usr/lib/perl5/5.18.1/x86_64-linux-thread-multi/CORE/libperl.so", "elfType" : 3, "buildId" : "217B4A7EA7DF2A7B46BC93993FD7630EC1175D0C" }, { "b" : "7F34A6EE1000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8EEA8B23F6DF125C66129A2DB29614A38300B77A" }, { "b" : "7F34A6CCA000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "799562EB36D9479DF11B4BB13D8C29D852484FE8" }, { "b" : "7F34A6A64000", "path" : "/usr/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "EF846533CF9AA9F40966FF8F2C1D120F1847BB45" }, { "b" : "7F34A685F000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "B6C7F251CBA059292F8892B74F4D2C167665C785" }, { "b" : "7F34A6624000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "355B60D4EFA51BB76FF03EB56A0986F555755453" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x23) [0x199c053]
 mongod(_ZN5mongo10logContextEPKc+0xC1) [0x192a511]
 mongod(_ZN5mongo13fassertFailedEi+0x86) [0x1912df6]
 mongod(_ZN5mongo7fassertEib+0x34) [0x10220c4]
 mongod(+0x133B46C) [0x173b46c]
 mongod(__wt_eventv+0x8AB) [0x216be7b]
 mongod(__wt_err+0x165) [0x216c1d5]
 mongod(__wt_panic+0x3B) [0x216cb9b]
 mongod(__wt_illegal_value+0x81) [0x216cc31]
 mongod(__wt_block_read_off+0x264) [0x207bdf4]
 mongod(__wt_bm_read+0x1D9) [0x207c0f9]
 mongod(__wt_bt_read+0x10F) [0x20965af]
 mongod(__wt_cache_read+0x14B) [0x209d87b]
 mongod(__wt_page_in_func+0xB7) [0x2099f67]
 mongod(+0x1CB670E) [0x20b670e]
 mongod(__wt_tree_walk+0x5C2) [0x20b6162]
 mongod(__wt_btcur_prev+0x49B) [0x208720b]
 mongod(+0x1CE158F) [0x20e158f]
 mongod(_ZN5mongo21WiredTigerRecordStore8Iterator7_locateERKNS_8RecordIdEb+0xCC) [0x174671c]
 mongod(_ZN5mongo21WiredTigerRecordStore8IteratorC2ERKS0_PNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionEb+0x172) [0x17465f2]
 mongod(_ZNK5mongo21WiredTigerRecordStore11getIteratorEPNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionE+0xD5) [0x1744cc5]
 mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextERKNS_10StringDataES5_bllPNS_28CappedDocumentDeleteCallbackEPNS_20WiredTigerSizeStorerE+0x499) [0x1740ad9]
 mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextERKNS_10StringDataES5_RKNS_17CollectionOptionsE+0x2EB) [0x173d36b]
 mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSs+0x157) [0x1699a87]
 mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x95F) [0x169f38f]
 mongod(+0x133A4F2) [0x173a4f2]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x1B7) [0x13339c7]
 mongod(+0xBBB16B) [0xfbb16b]
 mongod(_ZN5mongo13initAndListenEi+0x13) [0xfba0c3]
 mongod(+0xBBC243) [0xfbc243]
 mongod(main+0x2A) [0xfbbeba]
 libc.so.6(__libc_start_main+0xF5) [0x7f34a907bbe5]
 mongod(+0xBB9FE9) [0xfb9fe9]
-----  END BACKTRACE  -----
2014-12-18T15:22:44.911-0500 I -        [initandlisten] 
 
***aborting after fassert() failure


Generated at Thu Feb 08 03:41:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.