[SERVER-64145] The transaction is aborted even after stepdown or chunk migration is completed Created: 03/Mar/22 Updated: 06/Jun/23 Resolved: 06/Jun/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | beat jean | Assignee: | Max Hirschhorn |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Steps To Reproduce: | Tested on the following versions:
OS version: centos 7 Reproduce code:
WithTransactionExample function copy from https://docs.mongodb.com/manual/core/transactions/#transactions-api |
||||||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: |
| Description |
|
I have a shard cluster with 4000 shard collections. After executing stepdown on one of the shards, many errors will occur when executing transactions:
Similarly, after adding a shard to the shard cluster, many errors will occur when executing transactions:
For the latter case, `jstests/sharding/transactions_stale_shard_version_errors.js` explains that transaction failure is an expected behavior after chunk migration. And I can solve the above two problems by executing findOne (readpref is PrimaryMode) on each collection before executing the transaction after stepdown or chunk migration. So my questions and suggestions are: 1. Is it an expected behavior that the first transaction executed on each collection is aborted after the stepdown is complete?
|
| Comments |
| Comment by Max Hirschhorn [ 06/Jun/23 ] | |
|
We haven’t heard back from you for some time, so I'm going to close this ticket. If this is still an issue for you or if there are further questions, then please provide additional information and we will reopen the ticket. | |
| Comment by Max Hirschhorn [ 05/Dec/22 ] | |
|
It looks like you were already able to determine some of the system behaviors by reading through the server codebase and tests. Nice! I wanted to take a moment and clarify the answers to your initial questions and double check the overall behavior you are seeing.
Yes, for sharded collections and {$readPreference: {mode: "primary"}} is likely the first statement will trigger a stale shard version (StaleConfig, StaleEpoch, etc. exception). This is something which is true for both operations which happen within a transaction and operations which happen outside a transaction. The difference for operations within a transaction versus outside a transaction is whether mongos can automatically retry.
Would you clarify for me whether you are seeing the WithTransactionExample() return a non-nil error? Application errors would be unexpected due to the driver being capable of automatically retrying in this scenario. I'd like to make certain I have understood your question to be about the presence of the StaleConfig log messages and not due to an application error. Each member of the replica set shard tracks the sharding metadata for its collections in-memory independently. And so if a shard-versioned request has never been sent to the member of the replica set shard, then it won't have initialized the sharding metadata and will therefore need to first recover it from the config server.
Great question and there is some positive news in this direction. There are other ideas for having secondaries be more proactive the Sharding team is considering to explore. However if the secondary member never becomes the primary and is never targeted for reads either, then there was some wasted work involved and that's part of the tradeoff here. | |
| Comment by beat jean [ 26/Jul/22 ] | |
|
I found tickets related to this issue. SERVER-39624 states that if a transaction has a stale version error on some shards, then abort the transaction. And return TransientTransactionError to the client, the WithTransaction method in the driver will automatically retry the entire transaction for TransientTransactionError. But for the core api in the driver, it will not automatically retry, so there is still a problem here. SERVER-39704 looks like it should be dedicated to following up on this issue, but it doesn't seem to have been resolved.
| |
| Comment by beat jean [ 18/Jul/22 ] | |
|
Hi Chris Kelly, With some testing and code reading, I have some new progress. I found that the problem only occurs on newly started Mongod processes. After the StepDown is completed, if the newly started Mongod process becomes the primary, then the problem will occur. According to the code, the request sent by Mongos to Mongod has the correct shard version, and the newly started Mongod process regards all collections as UNSHARDED, and the two are inconsistent, so an error occurs in the check and the transaction is aborted. The newly started Mongod process treats all collections as unsharded because
is empty.
| |
| Comment by Chris Kelly [ 01/Jul/22 ] | |
|
We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket. | |
| Comment by Chris Kelly [ 10/Jun/22 ] | |
|
We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the requested information? Regards, | |
| Comment by Chris Kelly [ 17/May/22 ] | |
|
Thank you for your patience. To start answering your questions: 1) It doesn't appear that this is normal; in fact, there is an eerily similar error that occurred with the same workaround of doing a single find() in In order to get any more information, we'd need more log data to see what's going on. Could you please provide logs covering the following two events:
Would you please archive (tar or zip) the mongod.log files and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location? Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Regards, |