[SERVER-37730] Make the index catalog timestamp aware Created: 24/Oct/18  Updated: 06/Dec/22  Resolved: 16/Mar/20

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Dianna Hohensee (Inactive) Assignee: Backlog - Storage Execution Team
Resolution: Won't Do Votes: 0
Labels: execution_intern_2019
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Gantt Dependency
has to be done after SERVER-37843 Create an index reaper for two-phase ... Closed
Related
is related to SERVER-37733 Optimize rollback via refetch of drop... Closed
is related to SERVER-38548 Leverage the KVDropPendingIdentReaper... Closed
is related to SERVER-38797 Add unit tests for the IndexCatalog Closed
Assigned Teams:
Storage Execution
Sprint: Storage NYC 2019-01-14
Participants:

 Description   

Add 'birth' and 'death' timestamps to in-memory index catalog entries so that we can keep the in-memory index catalog entry objects and know when it is appropriate to use them. Handle standalones, too, they should default to zero timestamp for both 'birth' and 'death'. And built indexes loaded into memory can default to zero for the 'birth' timestamp as well, since PIT reads are not possible to before the index existed.
-----------
Delay dropping indexes until the index's death timestamp is checkpointed and older than supported point-in-time reads. Metadata changes in the storage engine act across all timestamps, they are not point-in-time.
------------
listIndexes should not return indexes with set 'death' timestamps.
------------
Two phase drop of indexes
1st Phase:

  • keep in-memory, set death timestamp
  • remove entry from __mdb_catalog
  • keep index table in WT
  • schedule table delete when death timestamp is older than both oldest_timestamp last checkpointed timestamp – see RTT explanation below for why.

2nd Phase (when death timestamp is old enough):

  • Remove in-memory entry
  • Delete table.

PIT accesses will see PIT version of __mdb_catalog that still has the index entry if the in-memory index entry allows index access at that PIT. Then the table still exists at that PIT and everything works.

Rollback via refetch works:

  • unschedule index table delete
  • rewrite __mdb_catalog entry
  • unset in-memory death timestamp

RTT works:

  • The in-memory catalog has been closed, so in-memory entries are of no concern
  • the __mdb_catalog entry will be recovered if appropriate on recovery to stable
  • if the __mdb_catalog entry was recovered, oplog application will reapply dropIndexes if it is not rolled back
  • easier to drop index table after dropIndexes is in the checkpoint, and will never be undone on rollback. Otherwise, the index table could be discarded, before dropIndexes is checkpointed, then recovery to the checkpoint will recreate the index catalog entry in __mdb_catalog and our hook to rebuild the index table will run – it'd be nicer if the index table wasn't dropped in the first place.

Question: does something bad happen if we checkpoint a timestamp that is newer than oldest timestamp? I imagine WT handles that situation, maybe checkpoints further back in time, earlier than oldest_timestamp. In case oldest_timestamp ever was set that far back in time, since the calculation doesn't currently refer to the checkpointed timestamp at all.



 Comments   
Comment by Connie Chen [ 16/Mar/20 ]

We'll be doing this another way

Comment by Benety Goh [ 16/Jul/19 ]

This ticket was originally intended to support the work in SERVER-38548, which defers ident drops for indexes.

Comment by Eric Milkie [ 25/Jan/19 ]

Temporarily putting work on this ticket on hold (not critical path work); may pick it up later.

Generated at Thu Feb 08 04:46:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.