[SERVER-60037] Enable the ordered timestamp assertion in MongoDB Created: 17/Sep/21  Updated: 29/Oct/23  Resolved: 15/Nov/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.2.0

Type: Improvement Priority: Major - P3
Reporter: Luke Pearson Assignee: Yuhong Zhang
Resolution: Fixed Votes: 0
Labels: CA-PM, post-mortem
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on WT-8297 timestamp order check fired on logged... Closed
Related
related to WT-8745 Restrict prepare_timestamp to be > st... Closed
related to SERVER-59831 WTUniqueIndex::_insert expects second... Closed
related to SERVER-60719 Hybrid index build side table uses mi... Closed
related to SERVER-60727 Remove mixed mode updates on local.sy... Closed
related to SERVER-60753 Removing index build entries from 'co... Closed
related to SERVER-60754 Setting multikey in _mdb_catalog can ... Closed
related to WT-8601 Timestamps do not work with logged ta... Closed
related to SERVER-63308 Accommodate WT-8601 Closed
related to WT-8170 Reduce complexity of timestamp usage ... Closed
is related to SERVER-58410 Discontinue writing to appliedThrough... Closed
Backwards Compatibility: Fully Compatible
Sprint: Execution Team 2021-10-18, Execution Team 2021-11-01, Execution Team 2021-11-15, Execution Team 2021-11-29
Participants:
Linked BF Score: 35

 Description   

It would be helpful for WiredTiger if we could turn on some assertions about timestamps, and as discussed in a recent meeting we should try to enable the mixed_mode timestamp assertion. I have attempted this and created a MongoDB patch build to check the fallout and there was still a significant amount. This is a shared ticket and will require a developer from the WiredTiger team and SERVER as fixing the test failures will be an iterative process.

When encountering a test failure a separate ticket should be created to fix the failure (if possible) and then this ticket will proceed once that ticket is completed, with the goal of one day enabling the assertion and merging it to master.

The patch I used to enable it was (it may not apply correctly so just take the config change):

diff --git a/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp b/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp
index 0a7d407ad6..f2bf8e5b2c 100644
--- a/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp
+++ b/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp
@@ -766,7 +766,7 @@ StatusWith<std::string> WiredTigerRecordStore::generateCreateString(
     // Choose a higher split percent, since most usage is append only. Allow some space
     // for workloads where updates increase the size of documents.
     ss << "split_pct=90,";
-    ss << "leaf_value_max=64MB,";
+    ss << "leaf_value_max=64MB,write_timestamp_usage=mixed_mode";
     ss << "checksum=on,";
     if (wiredTigerGlobalOptions.useCollectionPrefixCompression) {
         ss << "prefix_compression,";



 Comments   
Comment by Githook User [ 15/Nov/21 ]

Author:

{'name': 'Yuhong Zhang', 'email': 'danielzhangyh@gmail.com', 'username': 'YuhongZhang98'}

Message: SERVER-60037 Enable the ordered timestamp assertion in MongoDB
Branch: master
https://github.com/mongodb/mongo/commit/4cd66c0f2fe2912a5261d26bbc9608873fc22303

Comment by Githook User [ 20/Oct/21 ]

Author:

{'name': 'Yuhong Zhang', 'email': 'danielzhangyh@gmail.com', 'username': 'YuhongZhang98'}

Message: Revert "SERVER-60037 Enable the ordered timestamp assertion in MongoDB"

This reverts commit bf4cd96645253bb29a32ec0568095049bc49ab77.
Branch: master
https://github.com/mongodb/mongo/commit/5d54422313209ae5dc5be5de65450cce58a0ff00

Comment by Yuhong Zhang [ 19/Oct/21 ]

Instead of following the approach luke.pearson suggested, we skipped the checking on certain namespaces to enable the ordered timestamp assertion for the rest of the system first and filed the linked tickets to investigate the solutions to the existing issues.

Comment by Githook User [ 19/Oct/21 ]

Author:

{'name': 'Yuhong Zhang', 'email': 'danielzhangyh@gmail.com', 'username': 'YuhongZhang98'}

Message: SERVER-60037 Enable the ordered timestamp assertion in MongoDB
Branch: master
https://github.com/mongodb/mongo/commit/bf4cd96645253bb29a32ec0568095049bc49ab77

Comment by Luke Pearson [ 13/Oct/21 ]

yuhong.zhang the only difference for you is to change write_timestamp_usage=mixed_mode to write_timestamp_usage=ordered the other configuration option remain the same.

Comment by Luke Pearson [ 13/Oct/21 ]

I believe we were aiming lower as the "mixed-mode" assertion allows mixed mode updates while the ordered timestamp assertion does not. I am very happy to aim higher if it is achievable.

Comment by Daniel Gottlieb (Inactive) [ 13/Oct/21 ]

I changed the title to turning on the "ordered" timestamp assertion which I believe is a stronger guarantee as it disallows mixed-mode and out of order writes. luke.pearson, please reach out if this is not the case, or if we're intentionally shooting for a lower bar.

Comment by Yuhong Zhang [ 13/Oct/21 ]

Thanks to the pointers from gregory.noma, we suspect there might be some implementation details on multikey indexes that violate the in-order restriction. I managed to reproduce the issue with the following steps:
1. Start a replset
2. In a mongo shell run (the order of the indexes and documents matter)

db.test.createIndex({b: 1})
db.test.createIndex({a: 1})
db.test.insert([{a: [1, 2]}, {b: [1, 2]}])

Comment by Yuhong Zhang [ 01/Oct/21 ]

I uploaded a new patch with all three configurations turned on as well as an extra comma to fix the parsing errors, and I believe now it's more accurately showing the list of failing tests. 

A quick look at the failing tasks: the only failing tests are "jstests/aggregation/group_conversion_to_distinct_scan.js" and "jstests/core/sort_merge.js" under different suites, as well as "StorageTimestampTests" and "repl" in dbtests. All failing tests failed this check and printed out something similar to

"WiredTiger message","attr":\{"message":"Commit timestamp unexpected usage: committing a transaction that updates a value with an older timestamp ((1633112096, 1)) than is associated with the previous update ((1633112096, 2)) on a table configured for mixed mode ordering"}

There isn't any failures on Enterprise RHEL variants (maybe the configs were not setup there?).

Comment by Luke Pearson [ 27/Sep/21 ]

I'm unsure if the above patch build is valid however it turns out you should have minimum 3 configurations enabled and passed in the create config.
write_timestamp_usage=mixed_mode,assert=(write_timestamp=on),verbose=[write_timestamp]

Comment by Luke Pearson [ 17/Sep/21 ]

connie.chen, its not clear exactly which team in SERVER would be required, I think both replication and storage execution? Either way this ticket represents a lot of shared effort so keep that in mind for scheduling.

louis.williams has some context around this ticket if you need further clarification.

Generated at Thu Feb 08 05:48:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.