[SERVER-28660] memory leak of mongodb database with SSL support Created: 06/Apr/17  Updated: 16/Nov/21  Resolved: 14/Apr/17

Status: Closed
Project: Core Server
Component/s: Security
Affects Version/s: 3.4.2
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: John Wang [X] Assignee: Kelsey Schubert
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

The client is a Java program. Server: operating system: Ubuntu 16, CPU: 4 core 3.4GHZ, RAM:8GB, database: Mongodb 3.4 Community version


Attachments: Microsoft Word diagnostic_data.docx    
Issue Links:
Backports
Duplicate
duplicates SERVER-28530 SSLThreadInfo invokes constructor in ... Closed
Participants:

 Description   

We have Mongodb database server runs with SSL support, the config file about SSL is below, The certificate is self sign created with OpenSSL,
/etc/mongod.conf

net:
  port: 27017
#  bindIp: 127.0.0.1
  ssl:
    mode: requireSSL
    PEMKeyFile: /home/puma/atlas-ssl.pem

The clients is a Java multi thread program, write a record into Mongodb database every minute,

import java.util.Date;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
 
import org.bson.Document;
 
import com.mongodb.MongoClient;
import com.mongodb.MongoClientURI;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
 
public class PressureTest {
	public static void main(String[] args) {
		ExecutorService executor = Executors.newFixedThreadPool(100);
		for (int i = 0; i < 100; i++) {
			Runnable worker = new WorkerThread();
			executor.execute(worker);
		}
	}
}
 
class WorkerThread implements Runnable {
 
	@Override
	public void run() {
		System.setProperty("javax.net.ssl.trustStore",
				"C:/ssl/atlas-truststore.ts");
		System.setProperty("javax.net.ssl.trustStorePassword", "password");
		while (true) {
			insert();
			hangup();
		}
	}
 
	private void hangup() {
		try {
			Thread.sleep(60000);
		} catch (InterruptedException e) {
			e.printStackTrace();
		}
	}
 
	private void insert() {
		MongoClient client = null;
		try {
			if (client == null) {
				client = new MongoClient(new MongoClientURI(
						"mongodb://192.168.244.193:27017/atlas?ssl=true&sslInvalidHostNameAllowed=true"));
			}
			MongoDatabase db = client.getDatabase("atlas");
			MongoCollection<Document> collection = db.getCollection("operating_data");
				document.put("serialNumber", Thread.currentThread().getName());
			document.put("timestamp", new Date());
			document.put("FWver", "01.00.00");
			document.put("HWver", "03");
			document.put("harmonizerModel", "String");
			document.put("bypassTap1", "00.0");
			document.put("lowTap", "00.00");
			document.put("medTap", "00.00");
			document.put("highTap", "00.00");
			document.put("id", 0);
			document.put("msg", "Message string");
			collection.insertOne(document);
		} catch (Exception e) {
		} finally {
			if (client != null) {
				client.close();
			}
		}
	}
}

The issue is that the memory usage of Mongodb server keeps going up until the server throw out of memory error and mongodb service is collapsed.
Do you guys have any suggestion for this issue? I appreciate your help so much!

John



 Comments   
Comment by John Wang [X] [ 17/Apr/17 ]

Hi Ramon,

Thanks for your update so much! I would try the 3.4.4 release and give you feedback.

John

Comment by Ramon Fernandez Marina [ 17/Apr/17 ]

You're welcome John_Wang. Please note that we just made available a 3.4.4-rc0 release candidate, which should have the same bits as the upcoming 3.4.4 release. It's available for download here, if you install it and run it we'd love to hear whether it addresses your issue.

Thanks,
Ramón.

Comment by John Wang [X] [ 17/Apr/17 ]

Hi Ramon,

Thanks for your update so much!

John

Comment by Ramon Fernandez Marina [ 14/Apr/17 ]

MongoDB 3.4.4 is currently scheduled for the last week of April. You can subscribe to the mongodb-announce group if you're interested in receiving release notifications.

Thanks,
Ramón.

Comment by John Wang [X] [ 14/Apr/17 ]

Hi Thomas,

Thanks for your support so much!
When is MongoDB 3.4.4. gonna be released?

John

Comment by Kelsey Schubert [ 14/Apr/17 ]

Hi John_Wang,

Thank you for the additional information. After review, it appears that you are hitting SERVER-28530. A fix for this issue will be included in the upcoming release, MongoDB 3.4.4. Therefore, I would recommend upgrading to confirm that it resolves this issue.

Kind regards,
Thomas

Comment by John Wang [X] [ 13/Apr/17 ]

Hi Ramon,

Is there any update with this bug? Thanks so much for your support!

John

Comment by Ramon Fernandez Ext [ 08/Apr/17 ]

Test

Comment by John Wang [X] [ 07/Apr/17 ]

Hi Thomas,

The log file is 7.9 GB. I can`t upload it to here. Sorry for that!
The mongodb service in router server is collapsed in Mar 27, 8:15PM. In the log file, it shows that an "out of memory" error and there are over 600 connections are not closed.
Without SSL, the cluster works pretty good.
Please let me know what else do you need.

Thanks so much!

John

Comment by John Wang [X] [ 07/Apr/17 ]

Hi Thomas,

Thanks for reaching me so much! We use Mongodb sharded cluster consisting of 4 servers. One is router called router1, two of them are shard servers called data1 and data2. One if config server called config1.

The mongodb service is callapsed at the router server due to the memory leak.

Attached file has the output of linux top command of router and shard server. Please watch the swap, it was used even though the memory still has plenty of spare.
In addition, the file has the serverstatus information. Please watch the page eviction.

Thanks for your help so much!

John

Comment by Kelsey Schubert [ 07/Apr/17 ]

Hi John_Wang,

Thanks for the report, I'm sorry that your cluster has this instability. To help us continue investigate, would you please provide the complete log files and an archive of the diagnostic.data?

Thank you,
Thomas

Generated at Thu Feb 08 04:18:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.