[SERVER-76986] config server restart ( Out of memory) continuously after upgrading to mongodb-sharded:6.0.5 Created: 10/May/23  Updated: 28/Aug/23  Resolved: 28/Aug/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: achraf-CDG-CAPITAL bouaouda Assignee: Unassigned
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Hi , 

 

We have sharded cluster using bitnami helm chart deployment  : 

https://github.com/bitnami/charts/tree/main/bitnami/mongodb-sharded 

 

We just upgrade our cluster to 6.0.5 ( previous version is mongodb-sharded:4.4.1 ) .

 

before the upgrade : 

1- config server was stable with 2Gi memory 

 

After the upgrade ( create a new cluster and restore the dump to the new cluster )  :

1- the config server keeps consuming the memory then hitting the limit of memory capacity => config server pod ( deployed on k8s ) restart :

dmesg -T : 

[Wed May 10 10:34:28 2023] Memory cgroup out of memory: Kill process 29734 (conn2890) score 1615 or sacrifice child
[Wed May 10 10:34:28 2023] Killed process 186708 (mongod), UID 1001, total-vm:12369392kB, anon-rss:9388828kB, file-rss:33772kB, shmem-rss:0kB

mongo sys info : 
{
  system:

{     currentTime: ISODate("2023-05-10T10:55:30.141Z"),     hostname: 'mongo-distrib-prod-configsvr-0',     cpuAddrSize: 64,     memSizeMB: Long("23912"),     memLimitMB: Long("9216"),     numCores: 16,     numPhysicalCores: 16,     numCpuSockets: 8,     cpuArch: 'x86_64',     numaEnabled: false,     numNumaNodes: 1   }

,

2- increased from 2 Gi to 5Gi => 9Gi ( without success ) .

3- the same DB ( same version and helm chart )   is deployed on different ENV ( different workload ) and is working well 

Any suggestion please ?

 

 

Thank you 

 



 Comments   
Comment by Rachelle Palmer [ 28/Aug/23 ]

Hello there,
For triage and troubleshooting diagnostics, we recommend opening a case with our support team available at support.mongodb.com - our JIRA is for feature requests and bug reports.

Thank you,
Rachelle

Comment by achraf-CDG-CAPITAL bouaouda [ 10/May/23 ]

the issue fixed by removing metricbeat annotations ( disable metricbeat scraping ) :
annotations:
    co.elastic.logs/enabled: 'true'
    co.elastic.metrics/1.hosts: '${data.host}:27017'
    co.elastic.metrics/1.metricsets: 'status,collstats,dbstats,metrics,replstatus'
    co.elastic.metrics/1.module: mongodb
    co.elastic.metrics/1.password: xxx
    co.elastic.metrics/1.period: 10s
    co.elastic.metrics/1.username: xxxx
 
can you please analyze why?
 
Thank you

Generated at Thu Feb 08 06:34:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.