Container-write ops aren't ordering-correlated with their triggering CRUD op during oplog application

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Works as Designed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Execution
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      A container write is an oplog entry that writes to something other than a regular collection – it can be generated as a side effect of an ordinary document write (for example, when a document write also needs to update a secondary structure that isn't yet part of the collection's regular indexes). When that happens, the document's own CRUD op and the container op it produced are packed into the same applyOps entry, so a secondary can't apply one without the other.

      That grouping is only preserved up to the point the applyOps entry is unpacked for batch application. ApplyOps::extractOperations pulls the individual sub-ops back out of the entry, and each one is then independently assigned to a writer vector (the per-thread queue of ops a batch gets partitioned into) based on a hash: a CRUD op hashes on (collection, _id) via OplogApplierUtils::getOplogEntryHash, while a container op hashes on (container identity, raw key bytes) via getContainerKeyHash. These are unrelated hash domains, so the two ops can be assigned to different writer vectors – meaning different worker threads apply them, with no relative ordering between threads.

      There's already a mechanism that prevents exactly this: OplogApplierUtils::addDerivedOps takes a serial flag that, when true, assigns every extracted sub-op to the same writer vector instead of hashing each independently, which preserves their relative apply order. But the plain (non-transactional) applyOps extraction path – the one a grouped CRUD-op-plus-container-op entry goes through – hardcodes that flag to false. serial=true is only ever passed for prepared multi-statement transactions.

      Net effect: a document write and a container write derived from it, despite being correctly grouped into one applyOps entry at generation time, can be assigned to different writer vectors during batch application on a secondary, with no guarantee of which one applies first – even when both are in the same batch. Where the relative order between the two matters for correctness, that ordering isn't guaranteed.

      Suggested next steps

      • Determine whether serial should be extended to the plain applyOps extraction path the same way it already is for prepared transactions, or whether there's a reason this path is intentionally exempt.
      • If so, scope the fix and its cost – forcing a grouped entry's ops onto one writer vector reduces the parallelism available for any batch containing a container write.

            Assignee:
            Unassigned
            Reporter:
            Shin Yee Tan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: