[SERVER-42050] problem upgrading from 3.6.10 to 4.0.9 Created: 02/Jul/19  Updated: 29/Jul/19  Resolved: 29/Jul/19

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Lior Altarescu Assignee: Danny Hatcher (Inactive)
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

centos 7.4


Operating System: ALL
Participants:

 Description   

I was trying to upgrade a mongo replica set from version 3.2.12 to 4.0.9. i managed to upgrade all the server one at a time to 3.6.10 and had problems upgrading most of them to 4.0.9 .
the procedure in which i upgrade to 3.6.10 was to upgrade each secondary , after which failover. Upgrade the final server and then changed compatibility level . Between each upgrade I made sure that all the replicas were working find and checked that all compatibility level was correct after each upgrade.
while trying to upgrade the second server in the replica i got an error message :

2019-07-02T13:46:15.104+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2019-07-02T13:46:15.104+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2019-07-02T13:46:15.104+0000 I CONTROL [initandlisten]
2019-07-02T13:46:15.104+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2019-07-02T13:46:15.104+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2019-07-02T13:46:15.104+0000 I CONTROL [initandlisten]
2019-07-02T13:46:15.301+0000 F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.6 before attempting an upgrade to 4.0; see http://dochub.mongodb.org/core/4.0-upgrade-fcv for mor
e details.
2019-07-02T13:46:15.312+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2019-07-02T13:46:15.312+0000 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2019-07-02T13:46:15.312+0000 I REPL [initandlisten] shutting down replication subsystems
2019-07-02T13:46:15.312+0000 W REPL [initandlisten] ReplicationCoordinatorImpl::shutdown() called before startup() finished. Shutting down without cleaning up the replication system
2019-07-02T13:46:15.312+0000 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down
2019-07-02T13:46:15.313+0000 I STORAGE [initandlisten] WiredTigerKVEngine shutting down
2019-07-02T13:46:15.314+0000 I STORAGE [initandlisten] Shutting down session sweeper thread
2019-07-02T13:46:15.314+0000 I STORAGE [initandlisten] Finished shutting down session sweeper thread
2019-07-02T13:46:15.329+0000 I STORAGE [initandlisten] Downgrading WiredTiger datafiles.
2019-07-02T13:46:15.473+0000 I STORAGE [initandlisten] WiredTiger message [1562075175:473466][4603:0x7fe667aadb80], txn-recover: Main recovery loop: starting at 38514/118528 to 38515/256
2019-07-02T13:46:15.566+0000 I STORAGE [initandlisten] WiredTiger message [1562075175:566631][4603:0x7fe667aadb80], txn-recover: Recovering log 38514 through 38515
2019-07-02T13:46:15.622+0000 I STORAGE [initandlisten] WiredTiger message [1562075175:622052][4603:0x7fe667aadb80], txn-recover: Recovering log 38515 through 38515
2019-07-02T13:46:15.668+0000 I STORAGE [initandlisten] WiredTiger message [1562075175:668505][4603:0x7fe667aadb80], txn-recover: Set global recovery timestamp: 0
2019-07-02T13:46:15.910+0000 I STORAGE [initandlisten] shutdown: removing fs lock...
2019-07-02T13:46:15.910+0000 I CONTROL [initandlisten] now exiting
2019-07-02T13:46:15.910+0000 I CONTROL [initandlisten] shutting down with code:62
2019-07-02T13:46:15.942+0000 I CONTROL [main] ***** SERVER RESTARTED *****

i though that the problem was in syncing so I downgraded it to 3.6.10 and made sure that there was no replication lag but when I tried to reinstall 4.0.9 i got the same error. 

I tried to start the mongo with --repair but it failed twice so my only option is to run inital sync on those problematic servers, one at a time . 

I would for some help 
thanks



 Comments   
Comment by Danny Hatcher (Inactive) [ 29/Jul/19 ]

Closing due to lack of response.

Comment by Eric Sedor [ 16/Jul/19 ]

Hi,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide:

  • exact steps taken from the start of the upgrade process until the first error
  • the output of commands run as part of the process

Thanks,
Eric

Comment by Eric Sedor [ 03/Jul/19 ]

Hi lioral,

Can you please get the deployment to MongoDB 3.6 and provide the exact steps taken from the start of the upgrade process until the first error? If there is a bug in the upgrade process, we will need to reproduce it.

Importantly, please provide the output of commands run as part of the process.

Thanks in advance,
Eric

Generated at Thu Feb 08 04:59:24 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.