[SERVER-27131] While inserting records into MongoDB, the used memory of the instance running MongoDB keeps growing until it is shutdown Created: 20/Nov/16  Updated: 02/Dec/16  Resolved: 21/Nov/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.10, 3.2.11
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Haytham Salhi Assignee: Kelsey Schubert
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive diagnostic.data.zip    
Issue Links:
Duplicate
duplicates SERVER-23391 Improve behavior when running under W... Closed
Related
related to SERVER-22274 WiredTiger cache size is only configu... Closed
related to SERVER-23624 Allow wiredTigerCacheSizeGB to be spe... Closed
Operating System: ALL
Steps To Reproduce:
  • Run the mongoDB with its default configuration on an instance of 1GB RAM.
  • Insert records for a long period of time and monitor the memory of the instance.
Participants:

 Description   

I am running mongoDB server on an instance of 1 GB RAM. I have my client application on the other side inserting records for a somewhat long period of time (~ 1 day to 2 days).

What is happening is that the free memory of the instance keeps decreasing (or the used memory keeps increasing) until it reaches the maximum and seems the operating system kills the mongoDB.

I had two types of exceptions:

*The first one: *

org.springframework.dao.DataIntegrityViolationException: { "serverUsed" : "52.212.43.249:27017" , "ok" : 1 , "n" : 0 , "err" : "WiredTigerRecordStore::insertRecord 12: Cannot allocate memory" , "code" : 8}; nested exception is com.mongodb.WriteConcernException: { "serverUsed" : "52.212.43.249:27017" , "ok" : 1 , "n" : 0 , "err" : "WiredTigerRecordStore::insertRecord 12: Cannot allocate memory" , "code" : 8}
	at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:85)
	at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2114)
	at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:464)
	at org.springframework.data.mongodb.core.MongoTemplate.insertDBObject(MongoTemplate.java:1027)
	at org.springframework.data.mongodb.core.MongoTemplate.doInsert(MongoTemplate.java:838)
	at org.springframework.data.mongodb.core.MongoTemplate.insert(MongoTemplate.java:777)
	at org.springframework.data.mongodb.repository.support.SimpleMongoRepository.save(SimpleMongoRepository.java:78)
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:503)
	at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:488)
	at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:460)

or the second one (resulting from shutting down the mongoDB after the memory is exhausted)

org.springframework.dao.DataAccessResourceFailureException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=52.212.43.249:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused: connect}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=52.212.43.249:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.ConnectException: Connection refused: connect}}]
	at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:77)
	at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2114)
	at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:464)
	at org.springframework.data.mongodb.core.MongoTemplate.insertDBObject(MongoTemplate.java:1027)
	at org.springframework.data.mongodb.core.MongoTemplate.doInsert(MongoTemplate.java:838)
	at org.springframework.data.mongodb.core.MongoTemplate.insert(MongoTemplate.java:777)
	at org.springframework.data.mongodb.repository.support.SimpleMongoRepository.save(SimpleMongoRepository.java:78)



 Comments   
Comment by Kelsey Schubert [ 02/Dec/16 ]

Hi haytham.salhi,

We've included this change in behavior in our latest iteration of our documentation here: https://docs.mongodb.com/manual/reference/program/mongod/#cmdoption--wiredTigerCacheSizeGB. If you would like workarounds to be listed for the documentation for older versions of MongoDB, please feel free to open a DOCS ticket by clicking "report a problem" on the same page.

Kind regards,
Thomas

Comment by Haytham Salhi [ 21/Nov/16 ]

Thank you Thomas for the clarification. If such issue was previously reported, I think reflecting the solution (workarounds) on the manuals and documentation would be better as they are the first place where the developers go to when such issues appear, especially if solutions like the ones you mentioned are not indexed by search engines.

Anoter point to mention, I tried to specify a float value for wiredTigerCacheSizeGB (e.g. 0.5), but when attempting to start mongoDB, it does not run and the status remains 'stop'.

Best,
Haytham

Comment by Kelsey Schubert [ 21/Nov/16 ]

Hi haytham.salhi,

Thank you for providing the requested information. The output indicates that you are running 3.2.11 rather than 3.3.11 and I've updated the ticket to reflect that. In MongoDB 3.2.11, the default WiredTiger cache size is 1gb or 60% of available memory, whichever is larger. Additionally, it is expected that a system would run out of memory if the entirety of available memory was allocated to the WiredTiger cache.

In MongoDB 3.4, we have changed the default cache size to have a lower bound of 256mb to improve behavior when running WiredTiger on machines with little memory (SERVER-23391). In addition, we now allow wiredTigerCacheSizeGB to be specified as a float (SERVER-23624) to make configuration in these cases simpler.

Please review SERVER-22274 for workarounds to set the WiredTiger cache size to a float in MongoDB 3.0 and 3.2.

I'm closing this ticket since the required work has been completed in the linked tickets. For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

Kind regards,
Thomas

Comment by Haytham Salhi [ 21/Nov/16 ]

You might need to give attention to the files labeled with 19/11.

Comment by Haytham Salhi [ 20/Nov/16 ]

@Thomas: Attached is what you required.
@Dan: As mentioned in "Affects Version/s", the issue appeared while using the version 3.3.10, then I upgraded it to 3.3.11; the same issue though. Here is the output of mongod --version:

db version v3.2.11
git version: 009580ad490190ba33d1c6253ebd8d91808923e4
OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
allocator: tcmalloc
modules: none
build environment:
distmod: ubuntu1404
distarch: x86_64
target_arch: x86_64

Comment by Daniel Pasette (Inactive) [ 20/Nov/16 ]

Can you also please confirm the version of mongod you are using? Please post the output of mongod --version.

Comment by Kelsey Schubert [ 20/Nov/16 ]

Hi haytham.salhi,

Thanks for reporting this issue. So we can continue to investigate, would you please upload an archive of the diagnostic.data directory for an affected mongod?

Thank you,
Thomas

Comment by Haytham Salhi [ 20/Nov/16 ]

The instance is running Linux OS 14.04 LTS.

Generated at Thu Feb 08 04:14:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.