Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-8281

Fix out of order handling with history store stop timestamp

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT10.0.1, 5.2.0, 5.0.5
    • Affects Version/s: None
    • Component/s: None
    • None
    • 8
    • Storage - Ra 2021-11-01

      This is from a build failure in the internal testing.

      The following assert got hit with the given backtrace:

      file:_mdb_catalog.wt, txn rollback_to_stable: int __cdecl __rollback_ondisk_fixup_key(struct __wt_session_impl *,struct __wt_ref *,struct __wt_row *,unsigned __int64,struct __wt_item *,struct __wt_cell_unpack_kv *,unsigned __int64), 463: hs_stop_durable_ts <= newer_hs_durable_ts || hs_start_ts == hs_stop_durable_ts || hs_start_ts == newer_hs_durable_ts || newer_hs_durable_ts == hs_durable_ts || first_record || hs_stop_durable_ts == WT_TS_MAX
      

      i.e.

              WT_ASSERT(session,
                hs_stop_durable_ts <= newer_hs_durable_ts || hs_start_ts == hs_stop_durable_ts ||
                  hs_start_ts == newer_hs_durable_ts || newer_hs_durable_ts == hs_durable_ts ||
                  first_record || hs_stop_durable_ts == WT_TS_MAX);
      
      .../src/third_party/wiredtiger/src/os_common/os_abort.c@30: __wt_abort
      .../src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c@460: __rollback_ondisk_fixup_key
      .../src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c@776: __rollback_abort_ondisk_kv
      .../src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c@972: __rollback_abort_row_leaf
      .../src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c@1123: __rollback_abort_updates
      .../src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c@1206: __rollback_to_stable_btree_walk
      .../src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c@1553: __rollback_to_stable_btree_apply
      .../src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c@1648: __rollback_to_stable_btree_apply_all
      .../src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c@1714: __rollback_to_stable
      .../src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c@1754: __wt_rollback_to_stable
      .../src/third_party/wiredtiger/src/txn/txn_recover.c@1006: __wt_txn_recover
      .../src/third_party/wiredtiger/src/conn/conn_open.c@220: __wt_connection_workers
      

      Having looked at the core, the condition that has failed is hs_stop_durable_ts <= newer_hs_durable_ts.

      haribabu.kommi and I looked at the dump of the catalog file, for the affected key:

      K {36}
               value: len 383, start: (1634911810, 4)/(1634911810, 4)/0 stop: (1634911810, 4)/(1634911810, 4)/0
               V {\7f\01\00\00\03md\00\bb\00\00\00\02ns\00w\00\00\00db_1.long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_l    ong_ns_113_long_\00\03options\00 \00\00\00\05uuid\00\10\00\00\00\04\d6\8a\11\ef\07\05G\a4\b9\81H\c3\9c\dc\9b\94\00\04indexes\00\05\00\00\00\00\00\03idxIdent\00\05\    00\00\00\00\02ns\00w\00\00\00db_1.long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_\00\02ident\00"\    00\00\00collection-8--2768950964569208512\00\00}
               hs-update: start: (1634911790, 11)/(1634911790, 11)/0 stop: (1634911810, 4)/(1634911810, 4)/0
               V {8\02\00\00\03md\00M\01\00\00\02ns\00w\00\00\00db_1.long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_    ns_113_long_\00\03options\00 \00\00\00\05uuid\00\10\00\00\00\04\d6\8a\11\ef\07\05G\a4\b9\81H\c3\9c\dc\9b\94\00\04indexes\00\97\00\00\00\030\00\8f\00\00\00\03spec\0    0.\00\00\00\10v\00\02\00\00\00\03key\00\0e\00\00\00\10_id\00\01\00\00\00\00\02name\00\05\00\00\00_id_\00\00\08ready\00\01\08multikey\00\00\03multikeyPaths\00\10\00    \00\00\05_id\00\01\00\00\00\00\00\00\12head\00\00\00\00\00\00\00\00\00\08backgroundSecondary\00\00\00\00\00\03idxIdent\00,\00\00\00\02_id_\00\1d\00\00\00index-9--2    768950964569208512\00\00\02ns\00w\00\00\00db_1.long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_ns_113_long_\00    \02ident\00"\00\00\00collection-8--2768950964569208512\00\00}
               hs-update: start: (1634911790, 6)/(1634911790, 6)/0 stop: (1634911790, 17)/(1634911790, 17)/0
               V {u\01\00\00\02ns\00\0b\00\00\00db_0.three\00\03md\00\f8\00\00\00\02ns\00\0b\00\00\00db_0.three\00\03options\007\00\00\00\05uuid\00\10\00\00\00\04\aast\b1    \b3\d1B\84\9a\a5\9bU\06($v\08capped\00\01\12size\00\00\00\00\00\01\00\00\00\00\04indexes\00\97\00\00\00\030\00\8f\00\00\00\03spec\00.\00\00\00\10v\00\02\00\00\00\0    3key\00\0e\00\00\00\10_id\00\01\00\00\00\00\02name\00\05\00\00\00_id_\00\00\08ready\00\01\08multikey\00\00\03multikeyPaths\00\10\00\00\00\05_id\00\01\00\00\00\00\0    0\00\12head\00\00\00\00\00\00\00\00\00\08backgroundSecondary\00\00\00\00\00\03idxIdent\00+\00\00\00\02_id_\00\1c\00\00\00index-9-5221044324245339905\00\00\02ident\    00!\00\00\00collection-8-5221044324245339905\00\00}
               hs-update: start: (1634911790, 3)/(1634911790, 3)/0 stop: (1634911790, 6)/(1634911790, 6)/0
               V {\ae\01\00\00\03md\00\18\01\00\00\02ns\00$\00\00\00db_0.tmp8cxAW.convertToCapped.three\00\03options\00>\00\00\00\05uuid\00\10\00\00\00\04\aast\b1\b3\d1B\    84\9a\a5\9bU\06($v\08capped\00\01\12size\00\00\00\00\00\01\00\00\00\08temp\00\01\00\04indexes\00\97\00\00\00\030\00\8f\00\00\00\03spec\00.\00\00\00\10v\00\02\00\00    \00\03key\00\0e\00\00\00\10_id\00\01\00\00\00\00\02name\00\05\00\00\00_id_\00\00\08ready\00\01\08multikey\00\00\03multikeyPaths\00\10\00\00\00\05_id\00\01\00\00\00    \00\00\00\12head\00\00\00\00\00\00\00\00\00\08backgroundSecondary\00\00\00\00\00\03idxIdent\00+\00\00\00\02_id_\00\1c\00\00\00index-9-5221044324245339905\00\00\02n    s\00$\00\00\00db_0.tmp8cxAW.convertToCapped.three\00\02ident\00!\00\00\00collection-8-5221044324245339905\00\00}
      

      The first two HS updates have an overlapping range for the validity window. This should not have happened.

      I am creating this ticket to further investigate the root cause of the issue and to evaluate potential fallout.

            Assignee:
            haribabu.kommi@mongodb.com Haribabu Kommi
            Reporter:
            sulabh.mahajan@mongodb.com Sulabh Mahajan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: