[GODRIVER-2401] report error when connect to mongodb inside k8s Created: 30/Apr/22 Updated: 27/Oct/23 Resolved: 25/Oct/22 |
|
| Status: | Closed |
| Project: | Go Driver |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | pickup li | Assignee: | Matt Dale |
| Resolution: | Gone away | Votes: | 0 |
| Labels: | kubernetes | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
Summaryapplication write with go driver report error when connect to mongodb inside k8s, If change to directConnection=true will not report error but have performance issue. Please provide the version of the driver. If applicable, please provide the MongoDB server version and topology (standalone, replica set, or sharded cluster).go 1.18 require ( MongoDB server version: 5.0.6
How to ReproduceSteps to reproduce. If possible, please include a Short, Self Contained, Correct (Compilable), Example. We setup Mongodb's stand-alone architecture in k8s, and then used the , ] }_ The 192.168.65.5:53 here is the dns address inside k8s, and the I googled for a solution, and verified that using However, when I am doing multiple concurrent data acquisition or data I also tested c driver, no error will be reported without directConnect=true Additional BackgroundPlease provide any additional background information that may be helpful in diagnosing the bug. programs to test connect and performance issue ``` package main import ( "go.mongodb.org/mongo-driver/bson" var ( func init() { flag.StringVar(&srcUri, "srcuri", "mongodb://root:dbmotion#123@10.10.150.207:27717", "source mongodb uri") flag.StringVar(&srcDb, "srcdb", "db1", "srouce db") flag.StringVar(&srcColl, "srccoll", "t1", "srouce collection") flag.IntVar(&threadCount, "nt", 1, "goroutine count") flag.IntVar(&docCount, "ndoc", 1000000, "total docs to be inserted") flag.IntVar(&batchSize, "batch", 512, "insert batch size") flag.BoolVar(&pprof, "pprof", false, "start net/http/pprof") }func main() { fmt.Printf("%s %d threads insert into %s/%s.%s %d docs, batch size: %d\n", connOpt := options.Client().ApplyURI(srcUri) defer conn.Disconnect(nil) if err := conn.Ping(nil, nil); err != nil { fmt.Println(err) return } if err := conn.Database(srcDb).Collection(srcColl).Drop(nil); err != nil { fmt.Println(err) return }begin := time.Now() nDoc := docCount / threadCount var wg sync.WaitGroup for i := 0; i < threadCount; i++ { () go printStat(nIns, 10) if pprof { () elapse := time.Since(begin) totalMB := 1.0 * float64(totalDoc) * 10 / 1024 func insert(conn *mongo.Client, dbName string, collName string, nDoc int, batchSize int, nInserted *int) { var doc bson.D ) coll := conn.Database(dbName).Collection(collName) var docs []interface{} for i := 0; i < nDoc; i++ { *nInserted += len(docs) if len(docs) >= batchSize { *nInserted += len(docs) func nowStr() string { return time.Now().Format("2006-01-02 15:04:05") }func printStat(nIns []int, itvS int) { tiker := time.NewTicker(time.Duration(itvS) * time.Second) totalDoc := 0 fmt.Printf("%s all insert %d docs, %3.f doc/s, %.3f MB/s\n", nowStr(), ```
run the programs with './insert -srcuri="mongodb://root:dbmotion#123@10.10.150.208:27717/?directConnection=false" -nt 1 -trace'
|
| Comments |
| Comment by PM Bot [ 25/Oct/22 ] | |
|
There hasn't been any recent activity on this ticket, so we're resolving it. Thanks for reaching out! Please feel free to comment on this if you're able to provide more information. | |
| Comment by pickup li [ 03/Sep/22 ] | |
|
Thanks very much. I will test and check if it works | |
| Comment by Matt Dale [ 26/Aug/22 ] | |
|
Hey pickup112@gmail.com, thanks for the extra info! Based on that information, my theory about the server identifying itself as www-a1615-replica0-0-0.www-a1615-replica0-headless.qfusion-wcy-bm2:27017 is correct. Additionally, the topology appears to be a 3-node replica set and you are connecting directly to the "primary" node. MongoDB replica sets offer a configuration called "replica set horizons" that allow connecting to a MongoDB replica set using different DNS names. However, that feature is only supported when using one of the MongoDB Kubernetes Operators. See documentation for using replica set horizons with the the Enterprise operator here and with the Community operator here. Are you using one of the MongoDB Kubernetes Operators to run MongoDB on Kubernetes? If so, are you able to update the MongoDB Kubernetes service definition to add a replica set horizon? | |
| Comment by pickup li [ 09/Jul/22 ] | |
|
Yes,use mongo to connect to the instance like this: ``` root@www-a1615-replica0-0-0:/# mongo --host 10.10.88.123 --port 2635 -u user1 -p 'User1-123' --authenticationDatabase adminMongoDB shell version v4.4.13 ```
and db.runCommand({hello:1}) return: ``` www-a1615-replica0:PRIMARY> db.runCommand({hello:1}) , , , }, ```
| |
| Comment by Matt Dale [ 09/Jul/22 ] | |
|
Hey pickup112@gmail.com, are you able to connect to your MongoDB instance running in Kubernetes using either the mongosh command line tool or the legacy mongo command line tool? If so, please connect and run the following command and paste the output here:
As you noted earlier, the connection issue doesn't seem to be isolated to just the Go Driver, so I believe this is actually an issue with server configuration, not the Go Driver. Configuring MongoDB servers to run in Kubernetes is outside of my area of expertise, so if I'm not able to help you, I'll reassign this ticket. Thanks for your patience so far! | |
| Comment by pickup li [ 16/Jun/22 ] | |
|
Thanks for response.
| |
| Comment by Matt Dale [ 16/Jun/22 ] | |
|
Hey pickup112@gmail.com, thanks for the question and sorry about the slow response! I believe the problem is related to how MongoDB drivers update the server names based on the information in the initial server handshake. In this case, it seems like the server is identifying itself as mongo-a9f01-replica0-0-0.mongo-a9f01-replica0-headless.qfusion-admin:27017, so that's what the driver attempts to connect to. However, it also seems like that DNS name is not resolvable from the host where the client application is running. In directConnection=true mode, I believe that server name rewrite is not performed. I'm not sure why the performance is different when directConnection=true. I have some questions to help me understand more about what's going on:
| |
| Comment by pickup li [ 05/May/22 ] | |
|
I also tested the performance with the driver of c and java. Using directConnection=true, the single-thread performance will drop from 120MB/s to 60MB/s, and the 8-thread performance will drop from 600MB/s to 120MB/s |