[SERVER-78417] Mongod invariant while running bucket_timestamp_rounding.js with new fcv upgrade downgrade suite Created: 24/Jun/23  Updated: 29/Oct/23  Resolved: 13/Jul/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Adi Zaimi Assignee: Gregory Wlodarek
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Assigned Teams:
Storage Execution
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Execution NAMR Team 2023-07-24
Participants:
Linked BF Score: 135

 Description   

One of the patch builds resulted in a mongod invariant while running test bucket_timestamp_rounding.js under the new fcv_upgrade_downgrade_replica_sets_jscore_passthrough suite. 
The first few lines of the call stack is as follows:

[j6:prim] | 2023-06-24T03:39:12.929+00:00 I  CONTROL  31445   [conn31] "Frame","attr":{"frame":{"a":"7FE6F9154F7C","b":"7FE6F8A25000","o":"72FF7C","s":"_ZN5mongo12_GLOBAL__N_19callAbortEv","C":"mongo::(anonymous namespace)::callAbort()","s+":"3C"}}[j6:prim] | 2023-06-24T03:39:12.929+00:00 I  CONTROL  31445   [conn31] "Frame","attr":{"frame":{"a":"7FE6F915539A","b":"7FE6F8A25000","o":"73039A","s":"_ZN5mongo15invariantFailedEPKcS1_j","C":"mongo::invariantFailed(char const*, char const*, unsigned int)","s+":"40A"}}
[j6:prim] | 2023-06-24T03:39:12.929+00:00 I  CONTROL  31445   [conn31] "Frame","attr":{"frame":{"a":"7FE6CC56388D","b":"7FE6CC34D000","o":"21688D","s":"_ZN5mongo10timeseries14bucket_catalog8internal12removeBucketERNS1_13BucketCatalogERNS1_6StripeENS_8WithLockERNS1_6BucketENS2_11RemovalModeE","C":"mongo::timeseries::bucket_catalog::internal::removeBucket(mongo::timeseries::bucket_catalog::BucketCatalog&, mongo::timeseries::bucket_catalog::Stripe&, mongo::WithLock, mongo::timeseries::bucket_catalog::Bucket&, mongo::timeseries::bucket_catalog::internal::RemovalMode)","s+":"75D"}}[j6:prim] | 2023-06-24T03:39:12.930+00:00 I  CONTROL  31445   [conn31] "Frame","attr":{"frame":{"a":"7FE6CC55CFD4","b":"7FE6CC34D000","o":"20FFD4","s":"_ZN5mongo10timeseries14bucket_catalog8internal15closeOpenBucketERNS1_13BucketCatalogERNS1_6StripeENS_8WithLockERNS1_6BucketERSt6vectorINS1_12ClosedBucketESaISB_EE","C":"mongo::timeseries::bucket_catalog::internal::closeOpenBucket(mongo::timeseries::bucket_catalog::BucketCatalog&, mongo::timeseries::bucket_catalog::Stripe&, mongo::WithLock, mongo::timeseries::bucket_catalog::Bucket&, std::vector<mongo::timeseries::bucket_catalog::ClosedBucket, std::allocator<mongo::timeseries::bucket_catalog::ClosedBucket> >&)","s+":"144"}}[j6:prim] | 2023-06-24T03:39:12.930+00:00 I  CONTROL  31445   [conn31] "Frame","attr":{"frame":{"a":"7FE6CC55F766","b":"7FE6CC34D000","o":"212766","s":"_ZN5mongo10timeseries14bucket_catalog8internal8rolloverERNS1_13BucketCatalogERNS1_6StripeENS_8WithLockERNS1_6BucketERKNS2_12CreationInfoENS1_14RolloverActionE","C":"mongo::timeseries::bucket_catalog::internal::rollover(mongo::timeseries::bucket_catalog::BucketCatalog&, mongo::timeseries::bucket_catalog::Stripe&, mongo::WithLock, mongo::timeseries::bucket_catalog::Bucket&, mongo::timeseries::bucket_catalog::internal::CreationInfo const&, mongo::timeseries::bucket_catalog::RolloverAction)","s+":"236"}}[j6:prim] | 2023-06-24T03:39:12.930+00:00 I  CONTROL  31445   [conn31] "Frame","attr":{"frame":{"a":"7FE6CC55D979","b":"7FE6CC34D000","o":"210979","s":"_ZN5mongo10timeseries14bucket_catalog8internal16insertIntoBucketEPNS_16OperationContextERNS1_13BucketCatalogERNS1_6StripeENS_8WithLockEhRKNS_7BSONObjENS1_34CombineWithInsertsFromOtherClientsENS2_19AllowBucketCreationERNS2_12CreationInfoERNS1_6BucketE","C":"mongo::timeseries::bucket_catalog::internal::insertIntoBucket(mongo::OperationContext*, mongo::timeseries::bucket_catalog::BucketCatalog&, mongo::timeseries::bucket_catalog::Stripe&, mongo::WithLock, unsigned char, mongo::BSONObj const&, mongo::timeseries::bucket_catalog::CombineWithInsertsFromOtherClients, mongo::timeseries::bucket_catalog::internal::AllowBucketCreation, mongo::timeseries::bucket_catalog::internal::CreationInfo&, mongo::timeseries::bucket_catalog::Bucket&)","s+":"349"}}[j6:prim] | 2023-06-24T03:39:12.930+00:00 I  CONTROL  31445   [conn31] "Frame","attr":{"frame":{"a":"7FE6CC560029","b":"7FE6CC34D000","o":"213029","s":"_ZN5mongo10timeseries14bucket_catalog8internal6insertEPNS_16OperationContextERNS1_13BucketCatalogERKNS_15NamespaceStringEPKNS_10StringData19ComparatorInterfaceERKNS_17TimeseriesOptionsERKNS_7BSONObjENS1_34CombineWithInsertsFromOtherClientsENS2_19AllowBucketCreationENS1_16BucketFindResultE","C":"mongo::timeseries::bucket_catalog::internal::insert(mongo::OperationContext*, mongo::timeseries::bucket_catalog::BucketCatalog&, mongo::NamespaceString const&, mongo::StringData::ComparatorInterface const*, mongo::TimeseriesOptions const&, mongo::BSONObj const&, mongo::timeseries::bucket_catalog::CombineWithInsertsFromOtherClients, mongo::timeseries::bucket_catalog::internal::AllowBucketCreation, mongo::timeseries::bucket_catalog::BucketFindResult)","s+":"739"}}[j6:prim] | 2023-06-24T03:39:12.930+00:00 I  CONTROL  31445   [conn31] "Frame","attr":{"frame":{"a":"7FE6CC51E72B","b":"7FE6CC34D000","o":"1D172B","s":"_ZN5mongo10timeseries14bucket_catalog6insertEPNS_16OperationContextERNS1_13BucketCatalogERKNS_15NamespaceStringEPKNS_10StringData19ComparatorInterfaceERKNS_17TimeseriesOptionsERKNS_7BSONObjENS1_34CombineWithInsertsFromOtherClientsENS1_16BucketFindResultE","C":"mongo::timeseries::bucket_catalog::insert(mongo::OperationContext*, mongo::timeseries::bucket_catalog::BucketCatalog&, mongo::NamespaceString const&, mongo::StringData::ComparatorInterface const*, mongo::TimeseriesOptions const&, mongo::BSONObj const&, mongo::timeseries::bucket_catalog::CombineWithInsertsFromOtherClients, mongo::timeseries::bucket_catalog::BucketFindResult)","s+":"15B"}}
[j6:prim] | 2023-06-24T03:39:12.930+00:00 I  CONTROL  31445   [conn31] "Frame","attr":{"frame":{"a":"7FE6E7ADE381","b":"7FE6E7A00000","o":"DE381","s":"_ZN5mongo10timeseries23attemptInsertIntoBucketEPNS_16OperationContextERNS0_14bucket_catalog13BucketCatalogERKNS_15NamespaceStringEPKNS_10CollectionERNS_17TimeseriesOptionsERKNS_7BSONObjENS3_34CombineWithInsertsFromOtherClientsEb","C":"mongo::timeseries::attemptInsertIntoBucket(mongo::OperationContext*, mongo::timeseries::bucket_catalog::BucketCatalog&, mongo::NamespaceString const&, mongo::Collection const*, mongo::TimeseriesOptions&, mongo::BSONObj const&, mongo::timeseries::bucket_catalog::CombineWithInsertsFromOtherClients, bool)","s+":"1611"}}
[j6:prim] | 2023-06-24T03:39:12.930+00:00 I  CONTROL  31445   [conn31] "Frame","attr":{"frame":{"a":"7FE6E5CFAAC5","b":"7FE6E5BA1000","o":"159AC5","s":"_ZZN5mongo14write_ops_exec12_GLOBAL__N_123insertIntoBucketCatalogEPNS_16OperationContextEmmRKSt6vectorImSaImEEPS4_INS_9write_ops10WriteErrorESaISA_EEPbRKNS9_20InsertCommandRequestEENK4$_13clEm","C":"mongo::write_ops_exec::(anonymous namespace)::insertIntoBucketCatalog(mongo::OperationContext*, unsigned long, unsigned long, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<mongo::write_ops::WriteError, std::allocator<mongo::write_ops::WriteError> >*, bool*, mongo::write_ops::InsertCommandRequest const&)::$_13::operator()(unsigned long) const","s+":"B55"}}

Evergreen run: https://parsley.mongodb.com/resmoke/94129f6e20ded1bbd80d89fc40229911/test/176b7b34975308e5c5e2fa5da4a59b98?bookmarks=0,10902&shareLine=192



 Comments   
Comment by Githook User [ 13/Jul/23 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-78417 Set featureFlagTimeseriesAlwaysUseCompressedBuckets shouldBeFCVGated=false
Branch: master
https://github.com/mongodb/mongo/commit/3e2b98baa2306365ad50093657e48fb5f9ef31eb

Comment by Adi Zaimi [ 27/Jun/23 ]

I don't know if the previous comment was addressed to me, however I'm giving a bit of context of that is going on from the test side: 
we added a new testsuite as part of PM-3087 which sends continuously FCV upgrade/downgrade from a background hook. If you need any more info, please let me know.

Comment by Adi Zaimi [ 25/Jun/23 ]

It looks like timeseries_explicit_unpack_bucket.js also failed:
https://spruce.mongodb.com/task/mongodb_mongo_master_rhel80_debug_aubsan_all_feature_flags_fcv_upgrade_downgrade_sharded_collections_jscore_passthrough_1_linux_enterprise_patch_025efaf470f0dadfe925df8e9a9ef662cfa8522f_64977936562343e5c161218d_23_06_24_23_16_09/tests?execution=0&sortBy=STATUS&sortDir=ASC

with a very similar stack:

#2  0x00007f12e5065f7c in mongo::(anonymous namespace)::callAbort () at src/mongo/util/assert_util.cpp:76
#3  0x00007f12e506639a in mongo::invariantFailed (expr=0x7f12d1b8c760 <str> "!feature_flags::gTimeseriesAlwaysUseCompressedBuckets.isEnabled( serverGlobalParams.featureCompatibility)", file=0x7f12d1bbd044 "src/mongo/db/ops/write_ops_exec.cpp", line=2102) at src/mongo/util/assert_util.cpp:143
#4  0x00007f12d1c0d0c7 in mongo::invariantWithLocation<bool> (testOK=<optimized out>, expr=<optimized out>, file=<optimized out>, line=2102) at src/mongo/util/assert_util_core.h:74
#5  mongo::write_ops_exec::(anonymous namespace)::tryPerformTimeseriesBucketCompression (opCtx=<optimized out>, closedBucket=..., request=...) at src/mongo/db/ops/write_ops_exec.cpp:2101
#6  0x00007f12d1c0bf66 in mongo::write_ops_exec::(anonymous namespace)::insertIntoBucketCatalog(mongo::OperationContext*, unsigned long, unsigned long, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<mongo::write_ops::WriteError, std::allocator<mongo::write_ops::WriteError> >*, bool*, mongo::write_ops::InsertCommandRequest const&)::$_13::operator()(unsigned long) const (this=<optimized out>, index=<optimized out>) at src/mongo/db/ops/write_ops_exec.cpp:2471
#7  0x00007f12d1c075a8 in mongo::write_ops_exec::(anonymous namespace)::insertIntoBucketCatalog (opCtx=0x616001392080, start=0, numDocs=<optimized out>, indices=..., errors=0x7f125b7b83a0, containsRetry=0x7f125b7b8440, request=...) at src/mongo/db/ops/write_ops_exec.cpp:2481
#8  0x00007f12d1c06adf in mongo::write_ops_exec::(anonymous namespace)::performOrderedTimeseriesWritesAtomically (opCtx=<optimized out>, errors=<optimized out>, opTime=<optimized out>, electionId=<optimized out>, containsRetry=<optimized out>, request=...) at src/mongo/db/ops/write_ops_exec.cpp:2562
#9  0x00007f12d1bfdff4 in mongo::write_ops_exec::(anonymous namespace)::performOrderedTimeseriesWrites (opCtx=0x616001392080, errors=errors@entry=0x7f125b7b83a0, opTime=opTime@entry=0x7f125b7b83e0, electionId=electionId@entry=0x7f125b7b8420, containsRetry=containsRetry@entry=0x7f125b7b8440, request=...) at src/mongo/db/ops/write_ops_exec.cpp:2678
#10 0x00007f12d1bfd164 in mongo::write_ops_exec::performTimeseriesWrites (opCtx=<optimized out>, request=...) at src/mongo/db/ops/write_ops_exec.cpp:2736
#11 0x00007f12aadd3e78 in mongo::(anonymous namespace)::CmdInsert::Invocation::typedRun (this=0x616001392c80, opCtx=<optimized out>) at src/mongo/db/commands/write_commands.cpp:305
#12 0x00007f12aadd3612 in mongo::TypedCommand<mongo::(anonymous namespace)::CmdInsert>::InvocationBase::_callTypedRun (this=0x616001392c80, opCtx=0x616001392080) at src/mongo/db/commands.h:1324
 

Generated at Thu Feb 08 06:38:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.