Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
None

When walking the list of updates to store key/value pairs into the lookaside table, we encounter an illegal update type, WT_UPDATE_BIRTHMARK (0x1), causing an assert failure in mongod 4.2.

The corresponding failure occurs in the function '__wt_las_insert_block' (Line #729, File: src/cache/cache_las.c, https://github.com/wiredtiger/wiredtiger/blob/mongodb-4.2/src/cache/cache_las.c#L729). The following code snippet being:

switch (upd->type) {
            case WT_UPDATE_MODIFY:
            case WT_UPDATE_STANDARD:
                las_value.data = upd->data;
                las_value.size = upd->size;
                break;
            case WT_UPDATE_TOMBSTONE:
                las_value.size = 0;
                break;
            default:
                /*
                 * It is never OK to see a birthmark here - it would be referring to the wrong page
                 * image.
                 */
                WT_ERR(__wt_illegal_value(session, upd->type));

Associated mongod 4.2 stack trace:

[ReplicaSetFixture:job0:primary] ----- BEGIN BACKTRACE -----
[ReplicaSetFixture:job0:primary] {"backtrace":[{"b":"5606D5287000","o":"2A55281","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5606D5287000","o":"2A547BE"},{"b":"5606D5287000","o":"2A54856"},{"b":"7FE1DDDEE000","o":"11390"},{"b":"7FE1DDA24000","o":"35428","s":"gsignal"},{"b":"7FE1DDA24000","o":"3702A","s":"abort"},{"b":"5606D5287000","o":"D76490","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5606D5287000","o":"AE5E4E"},{"b":"5606D5287000","o":"115D78B"},{"b":"5606D5287000","o":"AF6FC9","s":"__wt_err_func"},{"b":"5606D5287000","o":"AF742D","s":"__wt_panic"},{"b":"5606D5287000","o":"11DA2C2","s":"__wt_las_insert_block"},{"b":"5606D5287000","o":"113EA20"},{"b":"5606D5287000","o":"1140656","s":"__wt_reconcile"},{"b":"5606D5287000","o":"1110DBC","s":"__wt_evict"},{"b":"5606D5287000","o":"1108DEA"},{"b":"5606D5287000","o":"1109695"},{"b":"5606D5287000","o":"110B8C9","s":"__wt_evict_thread_run"},{"b":"5606D5287000","o":"116B399"},{"b":"7FE1DDDEE000","o":"76BA"},{"b":"7FE1DDA24000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.12-44-g567b7b5", "gitVersion" : "567b7b59849dc2ecd10d6899b9b31c1aba5f7bd1", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "4.4.0-1030-aws", "version" : "#39-Ubuntu SMP Wed Aug 9 09:43:05 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "5606D5287000", "elfType" : 3, "buildId" : "ADA660049AE84DDD727C665A9CA12439FD094915" }, { "b" : "7FFF357C6000", "elfType" : 3, "buildId" : "1984636350CE171391CB4EB730718E599823C744" }, { "b" : "7FE1E08BD000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3D51E97C8DDD5F9EA1674954F8FA8BB2712E64B5" }, { "b" : "7FE1E06AE000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "E150CC35D09A20D93B4D936D72AA7637DABC978D" }, { "b" : "7FE1E0232000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpmibs.so.30", "elfType" : 3, "buildId" : "02D8231EB857EFBB37503B84D4A358A0A48EC4A1" }, { "b" : "7FE1E0023000", "path" : "/usr/lib/x86_64-linux-gnu/libsensors.so.4", "elfType" : 3, "buildId" : "1A855916EA4E082827D0599C11C0E82D98284118" }, { "b" : "7FE1DFE1F000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD" }, { "b" : "7FE1DFBB6000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "4206B1068C81403B1DB23AD71BD0E26A358BAC70" }, { "b" : "7FE1DF9AC000", "path" : "/lib/x86_64-linux-gnu/libwrap.so.0", "elfType" : 3, "buildId" : "54FCBC5B0F994A13A9B0EAD46F23E7DA7F7FE75B" }, { "b" : "7FE1DF6CF000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmp.so.30", "elfType" : 3, "buildId" : "EE53C5DD28800DF5A2F970B08F955A6A42B4BD41" }, { "b" : "7FE1DF28A000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "15FFEB43278726B025F020862BF51302822A40EC" }, { "b" : "7FE1DF06F000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "87783DF8A1058CD150F8886CB36340384093C18F" }, { "b" : "7FE1DEE25000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "41971A4A3CCDC54A447F41DF4BD96C948C546E0E" }, { "b" : "7FE1DEBB6000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "03F16B86D21AA657A636EB6D7977D285E5B111E4" }, { "b" : "7FE1DE8AD000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BAD67A84E56E73D031AE507261DA066B35949D34" }, { "b" : "7FE1DE692000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "50A923F8DAFECBCD969C8573116A38C18D0E24D5" }, { "b" : "7FE1DE429000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF69EA60EBE05F2DD689D2B26FC85A73E5FBC3A0" }, { "b" : "7FE1DE221000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "69143E8B39040C964D3958490535322675F15DD3" }, { "b" : "7FE1DE00B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FE1DDDEE000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B17C21299099640A6D863E423D99265824E7BB16" }, { "b" : "7FE1DDA24000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "1CA54A6E0D76188105B12E49FE6B8019BF08803A" }, { "b" : "7FE1E0B0E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E" }, { "b" : "7FE1DD7E3000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, { "b" : "7FE1DD4B3000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "213041D960C027C8C451F53B9B3143A901322015" }, { "b" : "7FE1DD2A6000", "path" : "/lib/x86_64-linux-gnu/libpci.so.3", "elfType" : 3, "buildId" : "7DAB2989A9974EF562D70AF1E8FEE0FB22E59966" }, { "b" : "7FE1DCECA000", "path" : "/usr/lib/x86_64-linux-gnu/libperl.so.5.22", "elfType" : 3, "buildId" : "E23A450DB91030D58256DBFF87C452400965E1D2" }, { "b" : "7FE1DCCB1000", "path" : "/lib/x86_64-linux-gnu/libnsl.so.1", "elfType" : 3, "buildId" : "3ABD85BB32116CB57BE934A30518021495C6A6EC" }, { "b" : "7FE1DC9DF000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "0EEF7058B0737B68BDF89E5DC604D0AC389C8BB1" }, { "b" : "7FE1DC7B0000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "FFBA483A43D9EF73925AC116811890C037523DA1" }, { "b" : "7FE1DC5AC000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "CBA6B89CE835324C6BEF5406AF1C6BC14D1205B3" }, { "b" : "7FE1DC3A1000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "B789D8D4B4FC333405AB34387D9237F954060EA4" }, { "b" : "7FE1DC16E000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "F3945D210B52CC973C619CCD08C4394BBC33E07F" }, { "b" : "7FE1DBF52000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, { "b" : "7FE1DBD38000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "8D9BD4CE26E45EF16075C67D5F5EEAFD8B562832" }, { "b" : "7FE1DBB2F000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, { "b" : "7FE1DB8A5000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, { "b" : "7FE1DB603000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, { "b" : "7FE1DB3D0000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, { "b" : "7FE1DB1BA000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, { "b" : "7FE1DAF56000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7FE1DAD43000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "A6EE7754728D688B04095ACB34F1A071F99ED6C3" }, { "b" : "7FE1DAB0D000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7FE1DA8DA000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7FE1DA65A000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7FE1E0CF2000", "path" : "/lib/x86_64-linux-gnu/libudev.so.1", "elfType" : 3, "buildId" : "8A39A962604C334F05A17CA205F7828938522FD2" }, { "b" : "7FE1DA422000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "FD61CA7A6D603E94E5EFD5C88D8810AE104FCF40" }, { "b" : "7FE1DA21E000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7FE1D9FF5000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, { "b" : "7FE1D9DE6000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, { "b" : "7FE1D9B9B000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, { "b" : "7FE1D98C6000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "DDCC7EE806C487FA3FA50C00F58E19CDEB902408" }, { "b" : "7FE1D96BE000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, { "b" : "7FE1D94B5000", "path" : "/usr/lib/x86_64-linux-gnu/sasl2/libgssapiv2.so", "elfType" : 3, "buildId" : "53588D90BAC52430A3841A5DA4F7E97A3992ADA9" }, { "b" : "7FE1D92AF000", "path" : "/usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so", "elfType" : 3, "buildId" : "8556E1F1222A0E264BB5FFFFF5540A44F5B4C20E" }, { "b" : "7FE1D90A8000", "path" : "/usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "6086975480D8FB099527D5BF2071AF1787440C1C" }, { "b" : "7FE1D8CFB000", "path" : "/usr/lib/x86_64-linux-gnu/libdb-5.3.so", "elfType" : 3, "buildId" : "58AD4D73AADCBF453B37E087F7567DBCAEEBACB8" }, { "b" : "7FE1D8AF2000", "path" : "/usr/lib/x86_64-linux-gnu/sasl2/libgs2.so", "elfType" : 3, "buildId" : "3DBCD75C273433C85455ADDC538092D7CC53BA73" }, { "b" : "7FE1D88ED000", "path" : "/usr/lib/x86_64-linux-gnu/sasl2/libplain.so", "elfType" : 3, "buildId" : "9C03E257E1B763E3EC7967C2B57EA83E1FAC496E" }, { "b" : "7FE1D86E4000", "path" : "/usr/lib/x86_64-linux-gnu/sasl2/libntlm.so", "elfType" : 3, "buildId" : "9A85CC9048DA2B9009C07949F118F6D08AC46969" }, { "b" : "7FE1D84DF000", "path" : "/usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "221552F99B28C6F78A39779E2D4A74601EFA26FC" }, { "b" : "7FE1D82D5000", "path" : "/usr/lib/x86_64-linux-gnu/sasl2/libscram.so", "elfType" : 3, "buildId" : "ADA7867764743267D4AB9B1E7A832B89DCFAA280" }, { "b" : "7FE1D80C7000", "path" : "/usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so", "elfType" : 3, "buildId" : "F1B50DB00C7DDC8615A5A8BDF80DF96141B0DF79" }, { "b" : "7FE1D7EC2000", "path" : "/usr/lib/x86_64-linux-gnu/sasl2/liblogin.so", "elfType" : 3, "buildId" : "C633C5DFE22DFFFBCB50892590D35839C4263CF7" } ] }}
[ReplicaSetFixture:job0:primary]  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5606d7cdc281]
[ReplicaSetFixture:job0:primary]  mongod(+0x2A547BE) [0x5606d7cdb7be]
[ReplicaSetFixture:job0:primary]  mongod(+0x2A54856) [0x5606d7cdb856]
[ReplicaSetFixture:job0:primary]  libpthread.so.0(+0x11390) [0x7fe1dddff390]
[ReplicaSetFixture:job0:primary]  libc.so.6(gsignal+0x38) [0x7fe1dda59428]
[ReplicaSetFixture:job0:primary]  libc.so.6(abort+0x16A) [0x7fe1dda5b02a]
[ReplicaSetFixture:job0:primary]  mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5606d5ffd490]
[ReplicaSetFixture:job0:primary]  mongod(+0xAE5E4E) [0x5606d5d6ce4e]
[ReplicaSetFixture:job0:primary]  mongod(+0x115D78B) [0x5606d63e478b]
[ReplicaSetFixture:job0:primary]  mongod(__wt_err_func+0x90) [0x5606d5d7dfc9]
[ReplicaSetFixture:job0:primary]  mongod(__wt_panic+0x39) [0x5606d5d7e42d]
[ReplicaSetFixture:job0:primary]  mongod(__wt_las_insert_block+0x332) [0x5606d64612c2]
[ReplicaSetFixture:job0:primary]  mongod(+0x113EA20) [0x5606d63c5a20]
[ReplicaSetFixture:job0:primary]  mongod(__wt_reconcile+0xC56) [0x5606d63c7656]
[ReplicaSetFixture:job0:primary]  mongod(__wt_evict+0x147C) [0x5606d6397dbc]
[ReplicaSetFixture:job0:primary]  mongod(+0x1108DEA) [0x5606d638fdea]
[ReplicaSetFixture:job0:primary]  mongod(+0x1109695) [0x5606d6390695]
[ReplicaSetFixture:job0:primary]  mongod(__wt_evict_thread_run+0x79) [0x5606d63928c9]
[ReplicaSetFixture:job0:primary]  mongod(+0x116B399) [0x5606d63f2399]
[ReplicaSetFixture:job0:primary]  libpthread.so.0(+0x76BA) [0x7fe1dddf56ba]
[ReplicaSetFixture:job0:primary]  libc.so.6(clone+0x6D) [0x7fe1ddb2b41d]
[ReplicaSetFixture:job0:primary] -----  END BACKTRACE  -----

[ReplicaSetFixture:job0:primary] 2021-02-22T04:57:51.837+0000 E  STORAGE  [thread539] WiredTiger error (22) [1613969871:837230][2903:0x7fe1d3eb9700], file:_mdb_catalog.wt, eviction-server: __wt_las_insert_block, 729: encountered an illegal file format or internal value: 0x1: Invalid argument Raw: [1613969871:837230][2903:0x7fe1d3eb9700], file:_mdb_catalog.wt, eviction-server: __wt_las_insert_block, 729: encountered an illegal file format or internal value: 0x1: Invalid argument
[ReplicaSetFixture:job0:primary] 2021-02-22T04:57:51.837+0000 E  STORAGE  [thread539] WiredTiger error (-31804) [1613969871:837281][2903:0x7fe1d3eb9700], file:_mdb_catalog.wt, eviction-server: __wt_panic, 489: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1613969871:837281][2903:0x7fe1d3eb9700], file:_mdb_catalog.wt, eviction-server: __wt_panic, 489: the process must exit and restart: WT_PANIC: WiredTiger library panic
[ReplicaSetFixture:job0:primary] 2021-02-22T04:57:51.837+0000 F  -        [thread539] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 486
[ReplicaSetFixture:job0:primary] 2021-02-22T04:57:51.837+0000 F  -        [thread539] \n\n***aborting after fassert() failure\n\n
[ReplicaSetFixture:job0:primary] 2021-02-22T04:57:51.924+0000 I  COMMAND  [conn492] command test1_fsmdb5.reindex_67 appName: "tid:67" command: getMore { getMore: 8873337223193855017, collection: "reindex_67", lsid: { id: UUID("0adf6cd6-6916-4f7d-b925-4c7afce06d39") }, $clusterTime: { clusterTime: Timestamp(1613969857, 70), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "test1_fsmdb5" } originatingCommand: { find: "reindex_67", filter: { $text: { $search: "ipsum" } }, lsid: { id: UUID("0adf6cd6-6916-4f7d-b925-4c7afce06d39") }, $clusterTime: { clusterTime: Timestamp(1613969857, 70), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "test1_fsmdb5" } planSummary: IXSCAN { _fts: "text", _ftsx: 1 } cursorid:8873337223193855017 keysExamined:899 docsExamined:899 cursorExhausted:1 numYields:0 nreturned:899 reslen:219482 locks:{ ReplicationStateTransition: { acquireCount: { w: 1 } }, Global: { acquireCount: { r: 1 } }, Database: { acquireCount: { r: 1 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 992257 } }, Collection: { acquireCount: { r: 1 } }, Mutex: { acquireCount: { r: 1 } } } storage:{} protocol:op_msg 1151ms
[ReplicaSetFixture:job0:primary] 2021-02-22T04:57:51.934+0000 F  -        [thread539] Got signal: 6 (Aborted).

is related to

WT-6518 WiredTiger attempts to insert a birthmark into lookaside (4.2)

Closed

Assignee:: Sulabh Mahajan

Reporter:: Alison Felizzi (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: Apr 19 2021 01:18:14 AM UTC

Updated:: Apr 27 2021 12:55:37 AM UTC

Resolved:: Apr 27 2021 12:54:42 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates