[SERVER-44594] Mongo Upgrade to 3.4 taking more time on Secondary Created: 13/Nov/19  Updated: 07/Apr/23  Resolved: 20/Nov/19

Status: Closed
Project: Core Server
Component/s: Upgrade/Downgrade
Affects Version/s: 3.4.23
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Pratiksha Aggarwal Assignee: Carl Champain (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

While upgrading mongodb from mongo 3.2 - mongo 3.4 on amazon linux machine mongo upgrade process on secondary servers is taking 20mins on each node, while on primary and read replica node  mongo upgrade process completed in 2min. 

The time took by upgrade process on secondary servers is 10mins while running transaction test and other 10min after transaction test was successful.

What can be the possible reason for this time?  The upgrade process was done on non-prod environment having half n hour of oplog window. On production environment on what basis can we predict the possible time for this process?? there can be the chances node going into recovery state on production environment if process takes an hour or so?



 Comments   
Comment by Carl Champain (Inactive) [ 20/Nov/19 ]

Hi pragupta,

Thanks for the report.

Unfortunately, we won't be able to investigate this issue as a bug since 3.2 is already end-of-lifed, and 3.4 will be end-of-lifed in January 2020.

I will now close this ticket. However, if you need further assistance troubleshooting, I encourage you to ask our community by posting on the mongodb-user group or on Stack Overflow with the mongodb tag.

Kind regards,
Carl

 

Comment by Pratiksha Aggarwal [ 18/Nov/19 ]

Please provide an update on this.

Comment by Pratiksha Aggarwal [ 15/Nov/19 ]

Sample Logs while mongo upgrade on secondary server

retrieving key from https://www.mongodb.org/static/pgp/server-3.4.asc
Importing GPG key ********:
Userid : "MongoDB 3.4 Release Signing Key <packaging@mongodb.com>"
Fingerprint: **************************************
From : https://www.mongodb.org/static/pgp/server-3.4.asc
Running transaction check
Running transaction test

      1. taking 10mins on secondary###
        Transaction test succeeded

 

    1. other 10mins for this as well

Running transaction
Updating : openldap-2.4.40-16.31.amzn1.x86_64 1/70
Updating : libcom_err-1.43.5-2.43.amzn1.x86_64 2/70
Updating : libuuid-2.23.2-59.29.amzn1.x86_64

Comment by Pratiksha Aggarwal [ 13/Nov/19 ]

We are using following sequence of commands for upgrading one node:

#shutting down mongo node

mongo $server:$portPri/$db -u $user -p $password < shutdownSecondary.js
echo "removing the pinned packages to let yum update to latest avaiable version"
sed -i.bak '/exclude=mongodb-org/d' /etc/yum.conf
echo "#Adding remote repo .list for mongodb version 3.4..."
echo "[mongodb-org-3.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.4.asc" | sudo tee /etc/yum.repos.d/mongodb-org-3.4.repo
echo "#Updating the yum repo to reflect changes done above..."
sudo yum -y update
echo "#Installing the mongodb version 3.4.17..."
sudo yum -y install mongodb-org-3.4.17 mongodb-org-server-3.4.17 mongodb-org-shell-3.4.17 mongodb-org-mongos-3.4.17 mongodb-org-tools-3.4.17
echo "#Pinning the mongodb version 3.4.17 to remove automatic updates to mongodb..."
echo "exclude=mongodb-org,mongodb-org-server,mongodb-org-shell,mongodb-org-mongos,mongodb-org-tools" >> /etc/yum.conf
echo "#Starting mongodb server.."

#starting the mongo process

mongod --verbose --config $configFileTer

Generated at Thu Feb 08 05:06:24 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.