[GODRIVER-951] golang connection consume huge memory, maybe memory leak Created: 10/Apr/19  Updated: 27/Oct/23  Resolved: 26/Apr/19

Status: Closed
Project: Go Driver
Component/s: Connections
Affects Version/s: 1.0.0-rc2
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: generalking Assignee: Kristofer Brandow (Inactive)
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

linux


Attachments: PNG File image-2019-04-19-15-44-09-746.png     PNG File image-2019-04-19-16-00-18-173.png     PNG File image-2019-04-19-16-04-41-580.png    

 Description   

when use mongo driver to connect several port parallel, the connect method use a lot of memory,

sample code: 

// 
type BaseServerStatus struct {
   Host           string    `bson:"host"`
   Version        string    `bson:"version"`
   Process        string    `bson:"process"`
   Pid            int64     `bson:"pid"`
   Uptime         int64     `bson:"uptime"`
   UptimeMillis   int64     `bson:"uptimeMillis"`
   UptimeEstimate int64     `bson:"uptimeEstimate"`
   LocalTime      time.Time `bson:"localTime"`
}
 
 
func GetBaseServerStatus(ip, port string) (srvStatus *BaseServerStatus, err error) {
   opts := options.Client()
   opts.SetDirect(true)
   opts.SetServerSelectionTimeout(1 * time.Second)
   opts.SetConnectTimeout(2 * time.Second)
   opts.SetSocketTimeout(2 * time.Second)
   opts.SetMaxConnIdleTime(1 * time.Second)
   opts.SetMaxPoolSize(1)
   url := fmt.Sprintf("mongodb://%s:%s/admin", ip, port)
   opts.ApplyURI(url)
   ctx, _ := context.WithTimeout(context.Background(), 2*time.Second)
   conn, err := mongo.Connect(ctx, opts)
   if err != nil {
      fmt.Printf("new %s:%s mongo connection error: %v\n", ip, port, err)
      return
   }
   defer conn.Disconnect(ctx)
   err = conn.Ping(ctx, nil)
   if err != nil {
      fmt.Printf("ping %s:%s ping error: %v\n", ip, port, err)
      return
   }
   sr := conn.Database("admin").RunCommand(ctx, bson.D{{"serverStatus", 1}})
   if sr.Err() != nil {
      fmt.Printf("get  %s:%s server status error: %v\n", ip, port, sr.Err())
      return
   }
   srvStatus = new(BaseServerStatus)
   err = sr.Decode(srvStatus)
   return
}
 
 
func main()  {
   var wg sync.WaitGroup
   //ips := []string{"xxx.xxx.xxx.xxx:22"}
   ips := []string{"xxx.xxx.xxx.xxx:22", "xxx.xxx.xxx.xxx:80", "xxx.xxx.xxx.xxx:7005", "xxx.xxx.xxx.xxx:7017", "xxx.xxx.xxx.xxx:7006", "xxx.xxx.xxx.xxx:7016", "xxx.xxx.xxx.xxx:7018", "xxx.xxx.xxx.xxx:7014", "xxx.xxx.xxx.xxx:199", "xxx.xxx.xxx.xxx:8182", "xxx.xxx.xxx.xxx:7015", "xxx.xxx.xxx.xxx:7022", "xxx.xxx.xxx.xxx:7013", "xxx.xxx.xxx.xxx:7020", "xxx.xxx.xxx.xxx:9009", "xxx.xxx.xxx.xxx:7004", "xxx.xxx.xxx.xxx:7008", "xxx.xxx.xxx.xxx:7002", "xxx.xxx.xxx.xxx:7021", "xxx.xxx.xxx.xxx:7007", "xxx.xxx.xxx.xxx:7024", "xxx.xxx.xxx.xxx:7010", "xxx.xxx.xxx.xxx:7011", "xxx.xxx.xxx.xxx:7003", "xxx.xxx.xxx.xxx:7012", "xxx.xxx.xxx.xxx:7009", "xxx.xxx.xxx.xxx:7019", "xxx.xxx.xxx.xxx:8001", "xxx.xxx.xxx.xxx:7023", "xxx.xxx.xxx.xxx:111", "xxx.xxx.xxx.xxx:7001", "xxx.xxx.xxx.xxx:8002", "xxx.xxx.xxx.xxx:19313", "xxx.xxx.xxx.xxx:15772", "xxx.xxx.xxx.xxx:19777", "xxx.xxx.xxx.xxx:15778", "xxx.xxx.xxx.xxx:15776"}
 
   for _, ip := range ips {
      wg.Add(1)
      //time.Sleep(3 * time.Second)
      go func(addr string) {
         fmt.Printf("start to probe port %s\n", addr)
         GetBaseServerStatus(strings.Split(addr, ":")[0], strings.Split(addr, ":")[1])
         wg.Done()
      }(ip)
   }
   wg.Wait()
   fmt.Println("scan end")
   time.Sleep(20 * time.Second)
 
}

**
when I use pprof to diagnose

the memory use like this

26.29GB 96.86% 96.86% 26.29GB 96.86% /go.mongodb.org/mongo-driver/x/network/connection.(*connection).ReadWireMessage



 Comments   
Comment by David Bartley [ 19/Jun/19 ]

GODRIVER-1152

Comment by David Bartley [ 19/Jun/19 ]

I dug around some more, and I think the original user's issue was triggered by, arguably, a bug in connction.ReadWireMessage. Specifically, that does:

    size := readInt32(sizeBuf[:], 0)
 
    // Isn't the best reuse, but resizing a []byte to be larger
    // is difficult.
    if cap(c.readBuf) > int(size) {
        c.readBuf = c.readBuf[:size]
    } else {
        c.readBuf = make([]byte, size)
    }

If you were to point the go driver at an HTTP server, it'd get back "HTTP" as the first 4 byes, a rather large 32-bit number when converted. You can even see in one of the attachments that go allocated a 1+ GiB block.
I believe mongod itself has a similar check for exactly this reason, and the go driver should probably also have this guard.

That said, I'm not seeing large blocks, only 16 MiB blocks, so I'll file a new ticket.

Comment by David Bartley [ 19/Jun/19 ]

We're also seeing this, though we're not doing anything particular odd like connecting to non-mongo ports. We just have a long-running process that tails the oplog. It's possible that the issue is tied to having a long-running cursor?

Comment by Kristofer Brandow (Inactive) [ 26/Apr/19 ]

generalking, this is not a supported use case for the MongoDB Go driver.

Comment by generalking [ 23/Apr/19 ]

I want to check whether a port is MongoDB or not.

Comment by Kristofer Brandow (Inactive) [ 22/Apr/19 ]

generalking If I understand your last comment correctly, you're using the MongoDB Go driver to connect to sshd and nginx. Can you elaborate on why you are doing this?

Comment by generalking [ 22/Apr/19 ]

No, I don't use benchmark suite in my test program. I just use the mongo driver to connect several ports, which are listened by sshd, nginx, python etc. 

Comment by Kristofer Brandow (Inactive) [ 19/Apr/19 ]

generalking, are you using the benchmark suite in the testing package? I can't reproduce your results. This is what top10 looks like for me:

 6754.88kB 35.36% 35.36%  6754.88kB 35.36%  go.mongodb.org/mongo-driver/x/bsonx.Doc.AppendMarshalBSON
 3584.57kB 18.77% 54.13%  3584.57kB 18.77%  go.mongodb.org/mongo-driver/x/bsonx/bsoncore.Document.Elements
 3081.14kB 16.13% 70.26%  3081.14kB 16.13%  go.mongodb.org/mongo-driver/x/bsonx.Doc.Append
 1056.33kB  5.53% 75.79%  2096.60kB 10.98%  go.mongodb.org/mongo-driver/x/network/connection.(*connection).ReadWireMessage
 1040.27kB  5.45% 81.24%  1040.27kB  5.45%  go.mongodb.org/mongo-driver/x/network/wiremessage.readDocument
  512.05kB  2.68% 83.92%   512.05kB  2.68%  go.mongodb.org/mongo-driver/x/network/command.ClientDoc
  512.05kB  2.68% 86.60%   512.05kB  2.68%  go.mongodb.org/mongo-driver/x/network/connection.NewPool
  512.02kB  2.68% 89.28%  7689.56kB 40.26%  go.mongodb.org/mongo-driver/x/bsonx.(*Val).UnmarshalBSONValue
  512.02kB  2.68% 91.96%   512.02kB  2.68%  go.mongodb.org/mongo-driver/x/bsonx.Document
  512.01kB  2.68% 94.64%   512.01kB  2.68%  errors.New 

Comment by generalking [ 19/Apr/19 ]

Comment by generalking [ 19/Apr/19 ]

Comment by generalking [ 19/Apr/19 ]

In my test program, i add this code 

go http.ListenAndServe(":8088", nil)

 

then, i run my test program

Comment by Kristofer Brandow (Inactive) [ 15/Apr/19 ]

generalking, Can you please provide more information about how your obtained that pprof output?

Generated at Thu Feb 08 08:35:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.