Reconcile checkpoint pick-up with metadata operations on the follower

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Engines
    • None
    • None

      We do not expect that picking up a checkpoint would create visible changes to the metadata (except for the initial checkpoint pick up), as the expectation is that the client's own replication is ahead of checkpoint pick up. However, it would be still worth finding and reconciling any discrepancies, or depending on the mode of execution, failing instead of reconciling differences. Reconciliation (as opposed to just checking) is likely needed for internal testing, but we can investigate if it could be split into a separate ticket, assuming that it is actually needed.

      We need to consider the following two cases:

      • If a table exists in the checkpoint's shared metadata table, but it does not exist locally, we would expect the metadata operation queue to have a published operation for dropping the table ahead of the checkpoint's checkpoint_schema_epoch, or an unpublished drop. If running in the checking mode, this would constitute a failure. If we would like to reconcile the changes, WiredTiger should pick up the new table from the shared metadata, as it means that the follower did not try to drop it since the checkpoint's schema epoch.
      • If a table exists in the local metadata but not in the checkpoint's shared metadata table, we would expect the metadata operation queue to contain an entry for creating the table that's either ahead of the checkpoint's checkpoint_schema_epoch or is not yet published. If we would like to reconcile the difference, we should do so by dropping the table locally, as this shows that the checkpoint's operation on that table is more recent than the client's operations on that table.

      Note that dropping a table in response to checkpoint pick up may result in EBUSY. A possible solution to this problem is to (1) create a queue of pending table drops, (2) process it at the next checkpoint pick up, (3) removing elements from it if the client creates a table with the same name locally, and (4) removing elements from it if the table appears in a subsequent checkpoint pick up. However, as this probably needs to be thought out a bit more carefully than that, we can move that work into a separate ticket.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Peter Macko
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: