[SERVER-54902] Time-series collection creation commits the view definition and associated oplog entry at different times Created: 03/Mar/21  Updated: 29/Oct/23  Resolved: 06/Apr/21

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 5.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Gregory Wlodarek Assignee: Henrik Edin
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-54668 Fail time-series collection creation ... Closed
Problem/Incident
causes SERVER-56286 Create timeseries bucket collection n... Closed
Related
related to SERVER-54899 Pass in point-in-time value used for ... Closed
related to SERVER-55909 Prevent a single WUOW from writing mu... Backlog
is related to SERVER-67736 Create time-series view and bucket co... Backlog
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Execution Team 2021-04-05, Execution Team 2021-04-19
Participants:
Linked BF Score: 22

 Description   

Time-series collection creation performs two operations under the hood:

  1. Create the buckets collection.
  2. Create the view definition on the buckets collection.

This all happens under one WriteUnitOfWork here. In that WUOW, there's an issue with the view definition creation part. Below are the order of operations for this WUOW to give a better idea.

1. start_transaction
2. setTimestamp(1)   <--- https://github.com/mongodb/mongo/blob/760ecda9059f3043498c2cd5106c8307344f756e/src/mongo/db/catalog/database_impl.cpp#L681 --->
3. createCollection (the buckets collection)
4. generate oplog entry via onCreateCollection at Timestamp(1)
5. Insert the view definition into the 'system.views' collection      <--- untimestamped --->
6. generate oplog entry via onInserts at Timestamp(2)
7. commit_transaction

Because the insert statement at (5) wasn't timestamped, it uses the last set commit timestamp, Timestamp(1) in this case. When the oplog entry was generated at (6), the oplog reserved the next available slot for the entry at Timestamp(2).

This means we committed the view definition at Timestamp(1) and its associated oplog entry at Timestamp(2).

 

 

A concurrency workload detected this issue when dbHash failed with a mismatch.
The oplog entries for the concurrency workload were the following:

1. T227 drop the view definition for the bucket (create_timeseries_collection_fsmcoll0_0)
2. T228 drop bucket (create_timeseries_collection_fsmcoll0_0)
3. T229 create bucket (create_timeseries_collection_fsmcoll0_0)
4. T230-T234 operations for another thread in the workload run   <--- this gap is important to detect the dbHash mismatch --->
5. T235 create the view definition for the bucket (create_timeseries_collection_fsmcoll0_0)

dbHash ran and opened a storage snapshot at T230 while the view definition should not have existed yet, but it did on the primary node. The secondary had the correct state though. Because of the issue described in this ticket, the view definition came into life at T229.

Primary running dbHash:
[ReplicaSetFixture:job0:primary] {"t":{"$date":"2021-03-02T20:47:09.435+00:00"},"s":"I",  "c":"COMMAND",  "id":0,       "ctx":"conn59","msg":"dbHash snapshot","attr":{"snapshot":"Timestamp(1614718029, 230)"}}
[ReplicaSetFixture:job0:primary] {"t":{"$date":"2021-03-02T20:47:09.435+00:00"},"s":"I",  "c":"COMMAND",  "id":0,       "ctx":"conn59","msg":"dbHash saw","attr":{"doc":{"_id":"test0_fsmdb0.create_timeseries_collection_fsmcoll0_0","viewOn":"system.buckets.create_timeseries_collection_fsmcoll0_0","pipeline":[{"$_internalUnpackBucket":{"timeField":"time","exclude":[]}}],"timeseries":{"timeField":"time"}}}}
[ReplicaSetFixture:job0:primary] {"t":{"$date":"2021-03-02T20:47:09.435+00:00"},"s":"I",  "c":"COMMAND",  "id":0,       "ctx":"conn59","msg":"dbHash saw","attr":{"doc":{"_id":"test0_fsmdb0.create_timeseries_collection_fsmcoll0_1","viewOn":"system.buckets.create_timeseries_collection_fsmcoll0_1","pipeline":[{"$_internalUnpackBucket":{"timeField":"time","exclude":[]}}],"timeseries":{"timeField":"time"}}}}
 
Secondary running dbHash:
[ReplicaSetFixture:job0:secondary] {"t":{"$date":"2021-03-02T20:47:09.436+00:00"},"s":"I",  "c":"COMMAND",  "id":0,       "ctx":"conn30","msg":"dbHash snapshot","attr":{"snapshot":"Timestamp(1614718029, 230)"}}
[ReplicaSetFixture:job0:secondary] {"t":{"$date":"2021-03-02T20:47:09.436+00:00"},"s":"I",  "c":"COMMAND",  "id":0,       "ctx":"conn30","msg":"dbHash saw","attr":{"doc":{"_id":"test0_fsmdb0.create_timeseries_collection_fsmcoll0_1","viewOn":"system.buckets.create_timeseries_collection_fsmcoll0_1","pipeline":[{"$_internalUnpackBucket":{"timeField":"time","exclude":[]}}],"timeseries":{"timeField":"time"}}}}



 Comments   
Comment by Githook User [ 05/Apr/21 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-54902 Creating timeseries bucket collection and view definition is now done in two separate WUOWs.

This should fix timestamping issue in the view definition document when it inherit a timestamp set by the collection creation code.
This should also allow users to fix the state where the bucket collection is majority replicated but the view definition document rolled back by running create again.
Branch: master
https://github.com/mongodb/mongo/commit/7380c412fb5cf7736828850186286d58e20215c9

Generated at Thu Feb 08 05:34:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.