[SERVER-22680] WiredTiger fails to start after trying to upgrade from 3.2.1 to 3.2.2 Created: 17/Feb/16  Updated: 04/Mar/16  Resolved: 04/Mar/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ronald Feicht Assignee: Ramon Fernandez Marina
Resolution: Duplicate Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongod.conf     File mongodb.tar.gz    
Issue Links:
Duplicate
duplicates SERVER-22676 WiredTiger fails to open databases cr... Closed
Operating System: ALL
Steps To Reproduce:

apt-get update && apt-get upgrade && apt-get dist-upgrade
press 'Y'

MongoDB data directory is attached - since it contains only test data there is no need to keep it secret.

Participants:

 Description   

After upgrading MongoDB to version 3.2.2 the database fails to start with the following error:

2016-02-17T08:55:21.482+0100 I CONTROL  [main] ***** SERVER RESTARTED *****
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten] MongoDB starting : pid=31130 port=27017 dbpath=/var/lib/mongodb 64-bit host=srv05
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten] db version v3.2.2
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten] git version: 6e71d0d568e134c029203593b00a0103e7cdf30b
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten] allocator: tcmalloc
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten] modules: none
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten] build environment:
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten]     distmod: debian71
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten]     distarch: x86_64
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten]     target_arch: x86_64
2016-02-17T08:55:21.492+0100 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1,192.168.2.83", port: 27017 }, processManagement: { fork: false }, setParameter: { enableLocalhostAuthBypass: "true" }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }
2016-02-17T08:55:21.523+0100 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-02-17T08:55:22.186+0100 E STORAGE  [initandlisten] WiredTiger (22) [1455695722:186444][31130:0x7f13f13bcd00], WT_SESSION.create: table:_mdb_catalog: does not match existing configuration: Invalid argument
2016-02-17T08:55:22.186+0100 I -        [initandlisten] Fatal Assertion 28562
2016-02-17T08:55:22.186+0100 I -        [initandlisten]
 
***aborting after fassert() failure

Downgrading to 3.2.1 or 3.2.0 did NOT fix it.
Happened only on our Ubuntu (Test) server, not on our Debian (Development) servers - the latter ones may have had their databases dropped more recently than the Ubuntu one.



 Comments   
Comment by Ramon Fernandez Marina [ 04/Mar/16 ]

Thanks for the update scsynergy, and glad to hear you were able to upgrade to 3.2.3. Closing this ticket now.

Cheers,
Ramón.

Comment by Ronald Feicht [ 18/Feb/16 ]

I upgraded to 3.2.3 without problems.

Comment by Ronald Feicht [ 18/Feb/16 ]

It seems that file versions from 3.2.2, 3.2.1 and 3.2.0 were mixed and garbled - I could not install any version due to errors. I guess all the upgrading, downgrading, removing, installing and purging at some point broke file version consistency. After I deleted all MongoDB files manually and then installed 3.2.1 I, too, can now start the database without problems. I do seem to remember that dpkg tried to start the database during an apt-get remove or purge or downgrade which failed, naturally, so I added 'exit 0' at the top of the init script to resolve that error. Could that have broken file version consistency? If so, you might want to change your packaging to not expect the database to start on package removal.

Anyway, thanks for helping even when the problem was on my end.

Comment by Ramon Fernandez Marina [ 17/Feb/16 ]

scsynergy, while I'm able to reproduce the issue with 3.2.2, if I downgrade to 3.2.1 I'm able to start mongod with the data files you sent without problems. I'm running Ubuntu 14.04 too, so I can't quite explain why a downgrade didn't work for you.

Can you please send the same startup log when you try with 3.2.1 and/or 3.2.0? This is what I get:

2016-02-17T11:44:19.076-0500 I CONTROL  [initandlisten] MongoDB starting : pid=28481 port=27017 dbpath=db 64-bit host=tab
2016-02-17T11:44:19.076-0500 I CONTROL  [initandlisten] db version v3.2.1
2016-02-17T11:44:19.076-0500 I CONTROL  [initandlisten] git version: a14d55980c2cdc565d4704a7e3ad37e4e535c1b2
2016-02-17T11:44:19.076-0500 I CONTROL  [initandlisten] allocator: tcmalloc
2016-02-17T11:44:19.077-0500 I CONTROL  [initandlisten] modules: none
2016-02-17T11:44:19.077-0500 I CONTROL  [initandlisten] build environment:
2016-02-17T11:44:19.077-0500 I CONTROL  [initandlisten]     distarch: x86_64
2016-02-17T11:44:19.077-0500 I CONTROL  [initandlisten]     target_arch: x86_64
2016-02-17T11:44:19.077-0500 I CONTROL  [initandlisten] options: { storage: { dbPath: "db" } }
2016-02-17T11:44:19.090-0500 I -        [initandlisten] Detected data files in db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-02-17T11:44:19.090-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=37G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-02-17T11:44:19.422-0500 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory 'db/diagnostic.data'
2016-02-17T11:44:19.422-0500 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-02-17T11:44:19.423-0500 I NETWORK  [initandlisten] waiting for connections on port 27017

Thanks,
Ramón.

Comment by Jacob Jewell [ 17/Feb/16 ]

This happened in our testing environment too. Luckily we caught it before it could corrupt our production databases.

Comment by Ramon Fernandez Marina [ 17/Feb/16 ]

Thanks for creating a new ticket scsynergy, we're investigating this issue now.

Generated at Thu Feb 08 04:01:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.