[SERVER-62457] Lock-free reads causes query subsystem to treat unsharded collection as sharded when collection is dropped and re-created (ABA problem) Created: 08/Jan/22  Updated: 29/Oct/23  Resolved: 10/Mar/22

Status: Closed
Project: Core Server
Component/s: Catalog
Affects Version/s: 5.1.1, 5.2.0-rc4
Fix Version/s: 6.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Dianna Hohensee (Inactive)
Resolution: Fixed Votes: 0
Labels: LFR-BUG
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Problem/Incident
is caused by SERVER-51319 Check DatabaseShardingState::checkDbV... Closed
Related
related to SERVER-76561 Lock-free acquisitions can read from ... Blocked
is related to SERVER-63009 The MigrationDestinationManager can s... Open
is related to SERVER-59178 Re-enable SBE as the default executio... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Execution Team 2022-01-24, Execution Team 2022-02-07, Execution Team 2022-02-21, Execution Team 2022-03-07, Execution Team 2022-03-21
Participants:
Linked BF Score: 19

 Description   

This can at least lead to a server crash with slot-based execution.

The query subsystem uses CollectionPtr::isSharded() to decide whether to add a plan stage to do ownership filter. As part of constructing the sbe::FilterStage to do ownership filter, the ShardFiltererImpl invariants the CollectionShardingState actually had a shard key pattern associated with it.

Constructing the special-purpose ShardFilterStage used by the classic executor doesn't have this invariant and simply forwards all documents through via kUnshardedCollection. This is why I believe only slot-based execution is affected.

The problematic sequence involves an unsharded collection being sharded and then dropped and then re-created as an unsharded collection:

  1. Collection exists and is unsharded.
  2. Mongos attaches shard version UNSHARDED to the request.
  3. Collection becomes sharded from some other client running shardCollection.
  4. AutoGetCollectionLockFree checks and sees the collection is now sharded.
  5. Collection is dropped from some other client running drop.
  6. Collection is created again as unsharded from some other client running create.
  7. AutoGetCollectionForReadCommandBase checks and sees the collection is unsharded. In particular, CollectionShardingState::checkShardVersionOrThrow() passes because the request was sent with shard version UNSHARDED and the collection is (again now) unsharded.

{"t":{"$date":"2022-01-08T01:39:33.733+00:00"},"s":"F",  "c":"ASSERT",   "id":23079,   "ctx":"conn23","msg":"Invariant failure","attr":{"expr":"_keyPattern","file":"src/mongo/db/exec/shard_filterer_impl.cpp","line":88}}

/home/ubuntu/mongo-v52/src/mongo/util/stacktrace_posix.cpp:263:40: LibunwindStepIteration
/home/ubuntu/mongo-v52/src/mongo/util/stacktrace_posix.cpp:434:36: mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) (.constprop.360)
/home/ubuntu/mongo-v52/src/mongo/util/stacktrace_posix.cpp:485:44: mongo::printStackTrace()
/home/ubuntu/mongo-v52/src/mongo/util/signal_handlers_synchronous.cpp:232:28: abruptQuit
??:0:0: ??
/build/glibc-S9d2JN/glibc-2.27/signal/../sysdeps/unix/sysv/linux/nptl-signals.h:80:0: __libc_signal_restore_set
/build/glibc-S9d2JN/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:48:0: gsignal
/build/glibc-S9d2JN/glibc-2.27/stdlib/abort.c:79:0: abort
/home/ubuntu/mongo-v52/src/mongo/util/assert_util.cpp:121:15: mongo::invariantFailed(char const*, char const*, unsigned int)
/home/ubuntu/mongo-v52/src/mongo/util/invariant.h:71:33: void mongo::invariantWithLocation<boost::optional<mongo::ShardKeyPattern> >(boost::optional<mongo::ShardKeyPattern> const&, char const*, char const*, unsigned int)
/home/ubuntu/mongo-v52/src/mongo/db/exec/shard_filterer_impl.cpp:88:35: mongo::ShardFiltererImpl::getKeyPattern() const
/home/ubuntu/mongo-v52/src/mongo/db/exec/shard_filterer_impl.cpp:87:19: mongo::ShardFiltererImpl::getKeyPattern() const (.cold.555)
/home/ubuntu/mongo-v52/src/mongo/db/query/sbe_stage_builder.cpp:2728:57: mongo::stage_builder::SlotBasedStageBuilder::buildShardFilter(mongo::QuerySolutionNode const*, mongo::stage_builder::PlanStageReqs const&)
...
/home/ubuntu/mongo-v52/src/mongo/db/query/sbe_stage_builder.cpp:2888:77: mongo::stage_builder::SlotBasedStageBuilder::build(mongo::QuerySolutionNode const*, mongo::stage_builder::PlanStageReqs const&)
/home/ubuntu/mongo-v52/src/mongo/db/query/sbe_stage_builder.cpp:695:45: mongo::stage_builder::SlotBasedStageBuilder::build(mongo::QuerySolutionNode const*)
/home/ubuntu/mongo-v52/src/mongo/db/query/stage_builder_util.cpp:76:47: mongo::stage_builder::buildSlotBasedExecutableTree(mongo::OperationContext*, mongo::CollectionPtr const&, mongo::CanonicalQuery const&, mongo::QuerySolution const&, mongo::PlanYieldPolicy*)
/home/ubuntu/mongo-v52/src/mongo/db/query/get_executor.cpp:881:62: buildExecutableTree
/home/ubuntu/mongo-v52/src/mongo/db/query/get_executor.cpp:647:18: prepare
/home/ubuntu/mongo-v52/src/mongo/db/query/get_executor.cpp:1100:52: mongo::(anonymous namespace)::getSlotBasedExecutor(mongo::OperationContext*, mongo::CollectionPtr const*, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::function<void (mongo::CanonicalQuery*)>, mongo::PlanYieldPolicy::YieldPolicy, unsigned long)
/home/ubuntu/mongo-v52/src/mongo/db/query/get_executor.cpp:1164:88: mongo::getExecutor(mongo::OperationContext*, mongo::CollectionPtr const*, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::function<void (mongo::CanonicalQuery*)>, mongo::PlanYieldPolicy::YieldPolicy, unsigned long)
/home/ubuntu/mongo-v52/src/mongo/db/query/get_executor.cpp:1190:38: mongo::getExecutorFind(mongo::OperationContext*, mongo::CollectionPtr const*, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::function<void (mongo::CanonicalQuery*)>, bool, unsigned long)
/home/ubuntu/mongo-v52/src/mongo/db/commands/find_cmd.cpp:548:69: mongo::(anonymous namespace)::FindCmd::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)



 Comments   
Comment by Githook User [ 09/Mar/22 ]

Author:

{'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@mongodb.com', 'username': 'DiannaHohensee'}

Message: SERVER-62457 Fix AutoGetCollectionForReadCommandLock bug wherein a request can come in with an UNSHARDED shard version, AutoGet*LockFree can find sharded state and then pass the SV check as unsharded, due to concurrent activities with the lock-free read state setup
Branch: master
https://github.com/mongodb/mongo/commit/0c242cb6ac5d61100cdcaee7f4533de7c2307d25

Comment by Dianna Hohensee (Inactive) [ 13/Jan/22 ]

On the other hand, I still think this is a bug, where we could have the ABA problem that Max describes.

AutoGet* helpers do the following
1) Fetch a Collection (here)
2) Fetch sharding state from the CollectionShardingState (here)
3) Fetch the Collection again to check that it is the same Collection without any updates (modifications or recreation) (here])
4) Check the shardVersion via CollectionShardingState (here])

0) request comes in for an unsharded collection
1) sharded collection is found
2) sharded state is found
3) sharded collection is found
4) collection has been dropped and recreated unsharded so shardVersion check passes
5) sharded read is consistent, but not what the caller was expecting

Musing: I suppose if an unsharded collection is queried, and it happens to be sharded and unsharded (dropped & recreated) before the mongod is locked, that can also run fine. Bit of a mystery read, but in a different way. Doesn't break the sharding protocol like lock-free does, though.

Comment by Dianna Hohensee (Inactive) [ 12/Jan/22 ]

For future safety against such uses, this seems to suggest lock-free reads must grab a copy of the CollectionShardingState (or whatever internal class) for the operation. Or add some invariants against accessing CollectionShardingState during a lock-free read – we have OperationContext::isLockFreeReadsOp().

Comment by Dianna Hohensee (Inactive) [ 12/Jan/22 ]

I think SBE is basically violating the original lock-free reads guarantees by accessing the CollectionShardingState directly instead of using what the AutoGet* helper sets up – for example, CollectionPtr::getShardKeyPattern(). The lock-free helpers must set up a point-in-time view of all in-memory and on-disk state that a query needs to access. SBE just bypassed it.

Max pointed out that there’s an implicit shardVersion check in the access query does to the CollectionShardingState. So if the first shardVersion check is correct, then subsequent accesses with version check should also be safe.

Comment by Eric Cox (Inactive) [ 11/Jan/22 ]

FYI geert.bosch max.hirschhorn I'm still looking into our usage of the shard key pattern and the interaction of SBE and forwarding all documents through via kUnshardedCollection. 

Generated at Thu Feb 08 05:55:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.