ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Add per-worker validator buffers and watermark/checkpoint bookkeeping directly to InflightBatch, alongside its existing multikey metadata. Give each work item a pointer to its worker's buffer for that batch, pass it into the apply function, and scope delta recording to the item so persistent workers cannot mix results from different batches. Keep those pointers valid as the batch moves through the tracker and until the advancer consumes its buffers. Evaluate validation enablement once per batch and classify watermark/checkpoint entries on the dispatcher. In _publishBatch(), record those entries and merge/validate the batch's deltas before multikey flushing and lastApplied publication. Unit test buffer isolation, lifetime, abandonment, cross-batch deltas, mismatch handling, and unchanged multikey behavior. Implement and exercise this path in the unit-test harness, but defer production activation until the baseline-adoption follow-up: mergeAndValidate() also adopts a baseline, which is unsafe without the adoption barrier.

      Some test coverage we should add:

      • Each worker records deltas into the correct batch-owned buffer, including when one persistent worker processes successive batches.
      • The thread-local buffer is set for each work item and cleared on exit, including exception paths. Write-conflict retries retain the correct buffer without recording contributions from rolled-back attempts.
      • Contributions from all workers are merged exactly once, with no missing or duplicate deltas; abandoned batches are never merged or published.
      • Grouped-insert fallback preserves every committed contribution without double-counting failed grouped attempts, and detects mismatched hashes when hash-capable coverage is available.
      • Later-completed batches cannot validate or publish before earlier batches, even when workers finish out of order.
      • Dispatcher classification of future batches does not mutate the current validator state.
      • Workers applying ahead across watermarks still contribute to the correct timestamp intervals, including hash intervals.
      • Replaying the same oplog with different batch boundaries produces equivalent checkpoint results.
      • Classic and POA replay produce equivalent checkpoint results across worker counts and batch boundaries. Extend this and the fallback/interval tests to hash correctness when ASC testing is available.

            Assignee:
            Ali Mir
            Reporter:
            Ali Mir
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: