-
Type: Bug
-
Resolution: Fixed
-
Priority: Critical - P2
-
Affects Version/s: 6.0.16, 7.3.3, 7.0.12, 8.0.0-rc7
-
Component/s: None
-
Catalog and Routing
-
Fully Compatible
-
ALL
-
v8.0, v7.3, v7.0, v6.0
-
CAR Team 2024-06-10, CAR Team 2024-06-24, CAR Team 2024-07-08, CAR Team 2024-07-22
ISSUE DESCRIPTION AND IMPACT
When cloning a Time Series collection, the internal timeseriesBucketsMayHaveMixedSchemaData flag is defaulted to false. If the user created a Time Series collection in v5.0-v5.1 and then upgraded to a later version before the collection was cloned, some queries may be missing matching documents.
All cloning procedures are incorrectly setting the timeseriesBucketsMayHaveMixedSchemaData flag: initial synchronization, chunk migration, movePrimary, resharding, mongodump/ mongorestore and others.
DIAGNOSIS
Users in v6.0+ versions can determine if they were impacted by running validate() on their Time Series collections and checking the validate.warnings field to determine if there are mixed-schema buckets detected.
The validation command can be very impactful. To minimize the performance impact of running validate, issue validate to a secondary following the hidden member steps outlined here.
- Example - validate run on a standalone/replica set:
// Call validate on a mongod process for replica sets. coll.validate(); // The warnings field detects mixed-schema buckets. { "ns" : "db.system.buckets.coll", ... "errors" : [ "Detected a Time Series bucket with mixed schema data when timeseriesBucketsMayHaveMixedSchemaData is false. You can run the collMod command to set this flag" ], ... }
- Example - validate run on a sharded cluster:
// Call validate on mongos for sharded clusters. coll.validate(); // The warnings field detects mixed-schema buckets. // For sharded clusters, this output is an object with a result for every shard in // the "raw" field. { "ns" : "db.system.buckets.coll", ... "errors" : [ "Detected a Time Series bucket with mixed schema data when timeseriesBucketsMayHaveMixedSchemaData is false. You can run the collMod command to set this flag" ], ... "raw" : { "shard-0-name" : { "ns" : "db.system.buckets.coll" ... "warnings" : [ "Detected a Time Series bucket with mixed schema data" ], ... }, "shard-1-name" : { ... }, ... } }
AFFECTED VERSIONS
The issue affects Time Series collections created before v5.2 that were later cloned.
The affected versions are: 6.0.0-6.0.16, 7.0.0-7.0.12, rapid releases 7.1-7.3.3. Later (sub-)versions are not affected.
Warning: Even if the cluster is currently on a safe version, any previous upgrade to an unsafe version in a cluster’s upgrade history means it may still be impacted.
REMEDIATION AND WORKAROUNDS
If impacted, upgrade o a fixed version and set the timeseriesBucketsMayHaveMixedSchemaData to true for each affected collection to ensure that future queries on the collection return correct results.
For example:
db.runCommand({ collMod: coll, timeseriesBucketsMayHaveMixedSchemaData: true });
Please see the remediation steps provided in our support-tools repo for a complete step-by-step guide on how to resolve the impacted collections.
Note: SERVER-91193 describes a related issue with the same root cause but does not require any manual intervention as it only impacts Rapid Release versions 7.1.0-7.3.3 and those clusters have been automatically upgraded.
Original description
The timeseriesBucketsMayHaveMixedSchemaData collection option is:
- Only set on time-series
- Serialized as top-level fields (not part of the options sub-object)
- Defaulted to false at bucket collection creation time
Defaulting timeseriesBucketsMayHaveMixedSchemaData to false (bullet 3) is problematic because collections are not only created from scratch but may also be created due to data cloning in the following cases:
- Initial synchronization - when cloning a collection as part of adding a new node to a replica set
- Chunk migration - when cloning a collection as part of migrating sharded collection's data
- MovePrimary - when cloning a collection as part of changing its db primary in a sharded cluster
- Resharding (and moveCollection) - when cloning a collection as part of redistributing all its data
- Mongodump/ mongorestore: when restoring a cluster from a backup
This means that in all clusters starting from v5.2 (SERVER-60574), the value of timeseriesBucketsMayHaveMixedSchemaData may be incorrect.
The short term solution is to always pretend the option is set to true (not changing the actual value on the catalog but always behave as if it was).
SERVER-91195 will take care of designing a long-term solution for avoiding hitting the issue in the future.
- depends on
-
SERVER-91195 Provide a generic backportable solution not to miss top-level timeseries collection options
- Closed
- is caused by
-
SERVER-60574 Add a new catalog flag to indicate whether a time-series bucket has mixed-schema data
- Closed
- is related to
-
SERVER-96831 Get rid of md.timeseriesBucketsMayHaveMixedSchemaData for avoiding confusion/misuses
- Backlog
-
SERVER-91969 Test initial sync with mixed-schema time-series buckets
- Closed
- related to
-
SERVER-84699 Add a check to validation to check for mixed-schema buckets for time-series
- Closed
-
SERVER-92441 Complete TODO listed in SERVER-91194
- Closed