[CSHARP-3301] Multi-thread Transaction Failure for Sharded Cluster Created: 07/Jan/21 Updated: 27/Oct/23 Resolved: 11/Jan/21 |
|
| Status: | Closed |
| Project: | C# Driver |
| Component/s: | Transactions |
| Affects Version/s: | 2.11.5 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | fini sky | Assignee: | Dmitry Lukyanov (Inactive) |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
MongoDB Sharded Cluster version: 4.2.2-ent |
||
| Description |
|
5 threads to execute transactions in parallel and encounter lots of 251 errors: MongoCommandException, 251, "NoSuchTransaction", "Command insert failed: cannot continue txnId 35 for session 618e6cd1-4db1-40ea-8b22-6386e204c36b - xxx with txnId 36"
Reproduce code:
If change the thread to 1, no error happens.
If not reuse the mongoClient by changing TestTransactionAsync(): create a dedicated mongoClient for each thread, no error happens:
The above modification intentionally passes different ServerSelectionTimeout value to prevent mongoclient from reusing. Refer to: https://mongodb.github.io/mongo-csharp-driver/2.11/reference/driver/connecting/#mongo-client
The document suggests re-use mongoclient by store it in a global place. However, a singleton mongoclient leads to parallel transaction failure. |
| Comments |
| Comment by Dmitry Lukyanov (Inactive) [ 11/Jan/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks finiskygarden@gmail.com for your report! | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by fini sky [ 11/Jan/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I found the root cause: the loadbalancer in front of mongos. Since there are 2 mongos instances behind the same stateless kubernetes service, a transaction may not be executed on the same mongos throughout its lifetime.
I'll expose every mongos instance seperately and change the connection string.
Thanks very much for Dmitry Lukyanov's kindly help! | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by fini sky [ 10/Jan/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Console app environment: .net core 2.2 (2.2.8) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by fini sky [ 09/Jan/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks Dmitry Lukyanov for your reply!
Log with 10 threads (driver 2.11.5, server 4.4.3-ent):
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Dmitry Lukyanov (Inactive) [ 08/Jan/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello finiskygarden@gmail.com . I've tried to reproduce the issue with your description and with no luck. Can you please specify/confirm the following details you can:
Please let me know if you have any questions. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Dmitry Lukyanov (Inactive) [ 07/Jan/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks finiskygarden@gmail.com for your report, we will investigate this and will let you know about results |