CPU usage leak on 'SECONDARY' node vesrions 7+

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: 7.0.26, 8.0.17
    • Component/s: None
    • ALL
    • Hide

      0 Step:

      Install the required tools to reproduce the issue

      docker
docker-compose

      1 Step:

      Create new RS with two data nodes and one arbiter. For fast reprudicing issue one data node arbiter will be unavalible, just broke resolve via '/etc/hosts'. For create topology run next bash code:

      docker-compose down
cat > docker-compose.yml << "EOF"
version: "3.9"

services:
  mongo1:
    image: mongo:7
    container_name: mongo1
    ports:
      - "27017:27017"
    command: >
      mongod --replSet rs0 --bind_ip_all

  mongo2:
    image: mongo:7
    container_name: mongo2
    command: >
      bash -c "echo '127.0.0.1 arbiter' >> /etc/hosts; mongod --replSet rs0 --bind_ip_all"
      
  arbiter:
    image: mongo:7
    container_name: arbiter
    command: >
      mongod --replSet rs0 --bind_ip_all --port 27018 
    ports:
      - "27018:27018"
EOF

docker-compose up -d

#Just a little wait 
sleep 10;

docker exec mongo1 mongosh --eval 'rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "mongo1:27017" },
    { _id: 1, host: "mongo2:27017" },
    { _id: 2, host: "arbiter:27018", arbiterOnly: true }
  ]
})'

      After commands new mongo cluster will be created where mongo1 will be 'PRIMARY' and mongo2 will be 'SECONDARY'.

      docker exec mongo1 mongosh --eval 'rs.status()'

      Step 2:

      Initial measurements of CPU and thread count on hosts mongo1 and mongo2:

      mongo1 CPU #1

      Command:

      docker exec mongo1 /bin/bash -c 'ps xa | grep mongod | grep -v grep | awk '"'"'{print $4}'"'"''

      Result:

      0:13

      mongo2 CPU #1

      Command:

      docker exec mongo2 /bin/bash -c 'ps xa | grep mongod | grep -v grep | awk '"'"'{print $4}'"'"''

      Result:

      0:38

      Wait 30 minuts:

      Step 2: Second measurements of CPU and thread count on hosts mongo1 and mongo2:

      mongo1 CPU #2

      Command:

      docker exec mongo1 /bin/bash -c 'ps xa | grep mongod | grep -v grep | awk '"'"'{print $4}'"'"''

      Result:

      0:31

      mongo2 CPU #2

      Command:

      docker exec mongo2 /bin/bash -c 'ps xa | grep mongod | grep -v grep | awk '"'"'{print $4}'"'"''

      Result:

      3:21

      Conclusion:

      On the SECONDARY host (mongo2), CPU usage is six times higher than on the PRIMARY (mongo1) even without any workload.
      CPU usage gradually decreases over time.

      Show
      0 Step: Install the required tools to reproduce the issue docker docker-compose 1 Step: Create new RS with two data nodes and one arbiter. For fast reprudicing issue one data node arbiter will be unavalible, just broke resolve via '/etc/hosts'. For create topology run next bash code: docker-compose down cat > docker-compose.yml << "EOF" version: "3.9" services: mongo1: image: mongo:7 container_name: mongo1 ports: - "27017:27017" command: > mongod --replSet rs0 --bind_ip_all mongo2: image: mongo:7 container_name: mongo2 command: > bash -c "echo '127.0.0.1 arbiter' >> /etc/hosts; mongod --replSet rs0 --bind_ip_all" arbiter: image: mongo:7 container_name: arbiter command: > mongod --replSet rs0 --bind_ip_all --port 27018 ports: - "27018:27018" EOF docker-compose up -d #Just a little wait sleep 10; docker exec mongo1 mongosh --eval 'rs.initiate({ _id: "rs0", members: [ { _id: 0, host: "mongo1:27017" }, { _id: 1, host: "mongo2:27017" }, { _id: 2, host: "arbiter:27018", arbiterOnly: true } ] })' After commands new mongo cluster will be created where mongo1 will be 'PRIMARY' and mongo2 will be 'SECONDARY'. docker exec mongo1 mongosh --eval 'rs.status()' Step 2: Initial measurements of CPU and thread count on hosts mongo1 and mongo2: mongo1 CPU #1 Command: docker exec mongo1 /bin/bash -c 'ps xa | grep mongod | grep -v grep | awk '"'"'{print $4}'"'"'' Result: 0:13 mongo2 CPU #1 Command: docker exec mongo2 /bin/bash -c 'ps xa | grep mongod | grep -v grep | awk '"'"'{print $4}'"'"'' Result: 0:38 Wait 30 minuts: Step 2: Second measurements of CPU and thread count on hosts mongo1 and mongo2: mongo1 CPU #2 Command: docker exec mongo1 /bin/bash -c 'ps xa | grep mongod | grep -v grep | awk '"'"'{print $4}'"'"'' Result: 0:31 mongo2 CPU #2 Command: docker exec mongo2 /bin/bash -c 'ps xa | grep mongod | grep -v grep | awk '"'"'{print $4}'"'"'' Result: 3:21 Conclusion: On the SECONDARY host (mongo2), CPU usage is six times higher than on the PRIMARY (mongo1) even without any workload. CPU usage gradually decreases over time.
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      In a topology with two data nodes and one arbiter, when the arbiter becomes unavailable, the SECONDARY data node begins to leak CPU resources.
      This issue reproduces in MongoDB 7.x and 8.x, but does not occur in MongoDB 6.x (it was discovered after upgrading from 6 to 7).

      During ~5 hours, the mongod process consumes the equivalent of two CPU cores:

        1. mongo2_top.png
          mongo2_top.png
          52 kB
        2. mongo1_top.png
          mongo1_top.png
          51 kB
        3. CPU_usage.jpeg
          CPU_usage.jpeg
          57 kB

            Assignee:
            Unassigned
            Reporter:
            Alexey Veselov
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: