Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-5856

Failure in txn_recover on 4.4 to 4.2 downgrade

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • 5
    • Storage Engines 2020-03-23

      When downgrading from MongoDB 4.4 to 4.2, an invalid access error occurs when trying to start up 4.2. This happens on a clean shutdown of 4.4 with FCV set to 4.2 prior to shutdown.

      Here is the error:

      2020-03-16T05:40:30.474+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=806 port=27017 dbpath=/data/db 64-bit host=stinks
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten] db version v4.2.3
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten] git version: 6874650b362138df74be53d366bbefc321ea32d4
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten] modules: none
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten] build environment:
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten]     distmod: ubuntu1804
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten]     distarch: x86_64
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
      2020-03-16T05:40:30.480+0000 I  CONTROL  [initandlisten] options: { replication: { replSet: "rs420" } }
      2020-03-16T05:40:30.480+0000 I  STORAGE  [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2020-03-16T05:40:30.481+0000 I  STORAGE  [initandlisten]
      2020-03-16T05:40:30.481+0000 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
      2020-03-16T05:40:30.481+0000 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
      2020-03-16T05:40:30.481+0000 I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31496M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
      2020-03-16T05:40:31.041+0000 I  STORAGE  [initandlisten] WiredTiger message [1584337231:41330][806:0x7f25f1821b00], txn-recover: Recovering log 2 through 4
      2020-03-16T05:40:31.041+0000 F  -        [initandlisten] Invalid access at address: 0
      2020-03-16T05:40:31.080+0000 F  -        [initandlisten] Got signal: 11 (Segmentation fault).
       0x55d90b4d4601 0x55d90b4d3dfe 0x55d90b4d3fdc 0x7f25efe5c890 0x55d909c21f6e 0x55d909c22758 0x55d909c0b525 0x55d909b4e455 0x55d909b4fcbf 0x55d909b8e3e4 0x55d909b93d02 0x55d909b43974 0x55d909bae154 0x55d909b30403 0x55d909b30ce6 0x55d909ad1b20 0x55d909b314a7 0x55d909ab12b7 0x55d909aacf3a 0x55d909a6aa9a 0x55d909a6c7cb 0x55d909a40033 0x55d90a26f65f 0x55d909a186fa 0x55d909a1bd5d 0x55d90999efc9 0x7f25efa7ab97 0x55d909a1730a 

      Backtrace:

      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"55D908CB5000","o":"281F601","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55D908CB5000","o":"281EDFE"},{"b":"55D908CB5000","o":"281EFDC"},{"b":"7F25EFE4A000","o":"12890"},{"b":"55D908CB5000","o":"F6CF6E"},{"b":"55D908CB5000","o":"F6D758","s":"__wt_block_buffer_to_addr"},{"b":"55D908CB5000","o":"F56525","s":"__wt_bm_read"},{"b":"55D908CB5000","o":"E99455","s":"__wt_bt_read"},{"b":"55D908CB5000","o":"E9ACBF"},{"b":"55D908CB5000","o":"ED93E4","s":"__wt_row_leaf_key_work"},{"b":"55D908CB5000","o":"EDED02","s":"__wt_row_search"},{"b":"55D908CB5000","o":"E8E974","s":"__wt_btcur_insert"},{"b":"55D908CB5000","o":"EF9154"},{"b":"55D908CB5000","o":"E7B403"},{"b":"55D908CB5000","o":"E7BCE6"},{"b":"55D908CB5000","o":"E1CB20","s":"__wt_log_scan"},{"b":"55D908CB5000","o":"E7C4A7","s":"__wt_txn_recover"},{"b":"55D908CB5000","o":"DFC2B7","s":"__wt_connection_workers"},{"b":"55D908CB5000","o":"DF7F3A","s":"wiredtiger_open"},{"b":"55D908CB5000","o":"DB5A9A","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_"},{"b":"55D908CB5000","o":"DB77CB","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb"},{"b":"55D908CB5000","o":"D8B033"},{"b":"55D908CB5000","o":"15BA65F","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE"},{"b":"55D908CB5000","o":"D636FA"},{"b":"55D908CB5000","o":"D66D5D"},{"b":"55D908CB5000","o":"CE9FC9"},{"b":"7F25EFA59000","o":"21B97","s":"__libc_start_main"},{"b":"55D908CB5000","o":"D6230A","s":"_start"}],"processInfo":{ "mongodbVersion" : "4.2.3", "gitVersion" : "6874650b362138df74be53d366bbefc321ea32d4", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-70-generic", "version" : "#79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55D908CB5000", "elfType" : 3, "buildId" : "3D6FEF629A2597C50A1266D937C15010B54188A7" }, { "b" : "7FFF6A1E7000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "ECA3EAC69ED80ED3850445DAA61BE16DAA989179" }, { "b" : "7F25F139E000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "1C6BC2C0699CE0F7E848CA0B267E0CF07553F6AB" }, { "b" : "7F25F1183000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F25F0CB8000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "68CECD8742C0C91CDC82709C3E52A8C9F9451FC5" }, { "b" : "7F25F0A2B000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "38AE5B4499DA422AA6D86BAED1902662DED5F730" }, { "b" : "7F25F0827000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F25F061F000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F25F0281000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F25F0069000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "DFEEFA01D550CA5E892223A232A3C8AFC85C3E66" }, { "b" : "7F25EFE4A000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F25EFA59000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F25F161D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F25EF834000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F25EF617000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "EE6E9462BA2491F4EE8C4E52C3323274A9366614" }, { "b" : "7F25EF3FB000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F25EF1ED000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F25EEFA2000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "00F419F64B0E70D8C5EEF7050369AA40B2A6E090" }, { "b" : "7F25EED50000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "70EEF126558D1559A0A4E334FB68E4E9AABE90CB" }, { "b" : "7F25EEB42000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "C14042EC7BD22B9A07D2C16563FE3C2606F52AB7" }, { "b" : "7F25EE925000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F25EE5A7000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "0E2784298E7D3F4D894FE130ACEFA77C3E624F72" }, { "b" : "7F25EE242000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "4EAB784840E5FF0508776BFE44D64E97851214A6" }, { "b" : "7F25EE00E000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F25EDDD8000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F25EDB57000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F25ED881000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "69FBCF425EE6DF03DE93B82FBC2FC33790E68A96" }, { "b" : "7F25ED64F000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "F400D5D643A7F9696DF0E6148FA99BEE6C1BDDF7" }, { "b" : "7F25ED44B000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "17107881DF65C66B4C6D38CAB37C285FA44663BD" }, { "b" : "7F25ED240000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "D78D71E8E016A534281B25B97CD7E5E9DB5FE00A" }, { "b" : "7F25ED025000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, { "b" : "7F25ECDE4000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F25ECAB5000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F25EC8A2000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F25EC69E000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F25EC495000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F25EC208000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F25EBF66000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F25EBD30000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F25EBB1A000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F25EB912000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F25EB6E9000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F25EB4DA000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F25EB290000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F25EAF87000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "A609DB077BDC3B54A8C64BBCF82C2B7BD1CC8B98" }, { "b" : "7F25EAD4F000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55d90b4d4601]
       mongod(+0x281EDFE) [0x55d90b4d3dfe]
       mongod(+0x281EFDC) [0x55d90b4d3fdc]
       libpthread.so.0(+0x12890) [0x7f25efe5c890]
       mongod(+0xF6CF6E) [0x55d909c21f6e]
       mongod(__wt_block_buffer_to_addr+0x18) [0x55d909c22758]
       mongod(__wt_bm_read+0x45) [0x55d909c0b525]
       mongod(__wt_bt_read+0x375) [0x55d909b4e455]
       mongod(+0xE9ACBF) [0x55d909b4fcbf]
       mongod(__wt_row_leaf_key_work+0x1F44) [0x55d909b8e3e4]
       mongod(__wt_row_search+0x1132) [0x55d909b93d02]
       mongod(__wt_btcur_insert+0xFD4) [0x55d909b43974]
       mongod(+0xEF9154) [0x55d909bae154]
       mongod(+0xE7B403) [0x55d909b30403]
       mongod(+0xE7BCE6) [0x55d909b30ce6]
       mongod(__wt_log_scan+0xCE0) [0x55d909ad1b20]
       mongod(__wt_txn_recover+0x3A7) [0x55d909b314a7]
       mongod(__wt_connection_workers+0x37) [0x55d909ab12b7]
       mongod(wiredtiger_open+0x23AA) [0x55d909aacf3a]
       mongod(_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_+0xBA) [0x55d909a6aa9a]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb+0x99B) [0x55d909a6c7cb]
       mongod(+0xD8B033) [0x55d909a40033]
       mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0x52F) [0x55d90a26f65f]
       mongod(+0xD636FA) [0x55d909a186fa]
       mongod(+0xD66D5D) [0x55d909a1bd5d]
       mongod(+0xCE9FC9) [0x55d90999efc9]
       libc.so.6(__libc_start_main+0xE7) [0x7f25efa7ab97]
       mongod(_start+0x2A) [0x55d909a1730a]
      -----  END BACKTRACE  ----- 

      The same error signature is happening in several tests here

            Assignee:
            luke.pearson@mongodb.com Luke Pearson
            Reporter:
            tammy.bailey@mongodb.com Tammy Bailey (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: